Taught by Dr. Zhao Han, Augmented Reality is a Fall 2022 course (CSCI 498A/598B) offered by the Department of Computer Science at the Colorado School of Mines.
|Course Code:||CSCI 498A/598B|
|Semester Year:||08/23/2022 – 12/15/2022|
|Class Meeting Times:||Tuesdays, Thursdays|
3:30pm – 4:45pm
|Class Locations:||Anderson Hall 330|
|Instructor:||Dr. Zhao Han (He/Him)|
|Instructor Office Location:||Brown Hall 280M|
|Instructor Office Hours:||Tuesdays, Thursdays|
10:00am – 11:00am
or by appointment
Teaching Assistant (TA)
|Teaching Assistant (TA):||Abdulelah Talea|
|TA Office Location:||CTLM 258 (subject to change)|
|TA Office Hours:||Mondays, 1:00-2:00pm|
or by appointment
- Software engineering (CSCI 306)
- Linear algebra (MATH 332)
Canvas Course Website
As part of the learning experience at the Colorado School of Mines, our class will be utilizing online learning resources and experiences through the Canvas learning management system: https://elearning.mines.edu/courses/42097 (When not logged in, this URL will direct you to the e-learning homepage).
In order to help build an inclusive community in this online learning environment, you are encouraged to
- upload your profile picture to Canvas,
- Photos should be similar to the photos taken for passports or state identification cards.
- set your pronouns (why?),
- and add a preferred name.
Welcome to Augmented Reality
The field of augmented reality (AR) is rapidly developing both in academia (with two conferences in the top 20 venues of human-computer interaction) and industry (e.g., related to Facebook’s Metaverse plan). I hope you will enjoy learning different aspects of AR in this course!
Augmented reality (AR) is the process of augmenting the real world with virtual content, typically with hand-held or head-mounted displays. This course will provide an overview of augmented reality, including applications, technical foundations (computer vision, optics, graphics, and human factors), and recent developments in the field. Projects will be emphasized, using popular software tools.
Augmented reality is a systems-thinking problem where it is helpful to have a good understanding of several areas, including computer vision, optics, graphics, and human-computer interaction. Projects will provide students with opportunities to gain a deeper understanding of these areas, as well as to build a portfolio for possible future careers in augmented reality.
- Introduction to augmented reality, Unity, and Vuforia
- Image targets and transformations
- Sounds, animations
- Visual coherence: lighting, shadows, occlusions
- Situated visualization
- Interaction techniques
- Plane finding
At the completion of the course, you will be able to:
- Define what is AR, compare different types of AR displays, and understand the capabilities and applications of AR
- Explain the graphics pipelines of AR systems, i.e., model, view, and perspective transformations
- Explain image target tracking computer vision algorithms and develop applications using the Vuforia engine framework, and understand nonvisual tracking methods using a variety of sensors
- Design, develop and deploy AR applications using the Unity framework, such as modeling, animating, and manipulating AR objects, simulating physics of real objects, adding sounds, ensuring visual coherence by adding lighting and occlusions, adding virtual buttons, annotating physical objects, and deploying to smartphones
- Develop situated visualizations, understand the pipeline, challenges, x-ray visualization to reveal hidden objects, and select AR visualization techniques to declutter a scene,
- Select input and output modalities for AR object interactions and user interfaces
The assessments for this course include the following:
|Three programming assignments||50%|
|Short homework exercises||15%|
|Technical topic investigation*||10%|
*For graduate students the requirements are more challenging for the final project and the technical topic investigation. This is the main differentiating factor between the undergraduate and graduate versions of this class.
Table of Contents
About the Authors xxvii
Chapter 1: Introduction to Augmented Reality 1
Definition and Scope 3
A Brief History of Augmented Reality 4
Related Fields 28
Chapter 2: Displays 33
Multimodal Displays 34
Visual Perception 39
Requirements and Characteristics 40
Spatial Display Model 56
Visual Displays 58
Chapter 3: Tracking 85
Tracking, Calibration, and Registration 86
Coordinate Systems 87
Characteristics of Tracking Technology 90
Stationary Tracking Systems 96
Mobile Sensors 99
Optical Tracking 105
Sensor Fusion 117
Chapter 4: Computer Vision for Augmented Reality 121
Marker Tracking 123
Multiple-Camera Infrared Tracking 132
Natural Feature Tracking by Detection 138
Incremental Tracking 149
Simultaneous Localization and Mapping 156
Outdoor Tracking 164
Chapter 5: Calibration and Registration 179
Camera Calibration 180
Display Calibration 183
Chapter 6: Visual Coherence 195
Photometric Registration 205
Common Illumination 216
Diminished Reality 227
Camera Simulation 231
Stylized Augmented Reality 236
Chapter 7: Situated Visualization 239
Visualization Registration 245
Annotations and Labeling 248
X-Ray Visualization 254
Spatial Manipulation 260
Information Filtering 265
Chapter 8: Interaction 271
Output Modalities 272
Input Modalities 279
Tangible Interfaces 286
Virtual User Interfaces on Real Surfaces 294
Augmented Paper 295
Multi-view Interfaces 297
Haptic Interaction 304
Multimodal Interaction 304
Conversational Agents 306
Chapter 9: Modeling and Annotation 311
Specifying Geometry 312
Specifying Appearance 317
Semi-automatic Reconstruction 319
Free-Form Modeling 322
Chapter 10: Authoring 329
Requirements of AR Authoring 330
Elements of Authoring 333
Stand-Alone Authoring Solutions 335
Plug-In Approaches 339
Web Technology 341
Chapter 11: Navigation 345
Foundations of Human Navigation 346
Exploration and Discovery 347
Route Visualization 347
Viewpoint Guidance 350
Multiple Perspectives 354
Chapter 12: Collaboration 361
Properties of Collaboration Systems 362
Co-located Collaboration 364
Remote Collaboration 370
Chapter 13: Software Architectures 379
AR Application Requirements 380
Software Engineering Requirements 382
Distributed Object Systems 385
Scene Graphs 395
Developer Support 400
Chapter 14: The Future 409
What May Drive Business Cases 410
An AR Developer’s Wish List 411
Taking AR Outdoors 415
Interfacing with Smart Objects 417
Confluence of Virtual Reality and Augmented Reality 418
Augmented Humans 419
AR as a Dramatic Medium 420
AR as a Social Computing Platform 421
Policies and Campus Resources
The COVID Smart Classroom
As new variants of COVID-19 continue to evolve, all campus community members are asked to make thoughtful choices about their health and be mindful that those choices will affect our whole community.
- Anyone experiencing COVID-19 symptoms should wear a mask, not report to work or attend classes, and get tested as soon as possible.
- Anyone experiencing respiratory symptoms – even after a negative COVID test – should remember there are many respiratory viruses circulating in our community. Please be considerate of others and wear a mask whenever you have any cold-like symptoms.
- Masking is one of the most effective ways to protect yourself and others, especially in indoor settings when community transmission levels are high. Please continue to respect an individual’s decision to wear a mask even if it is not required. Masks are still required in the Student Health Center.
- Wash your hands frequently using soap and water or hand sanitizer.
If you test positive for COVID-19 (rapid antigen or PCR test):
- Stay home for a minimum of five (5) days and isolate yourself from others at home, as directed by the CDC.
- Communicate with your professors via email or the excused absence form that you are ill or have tested positive for COVID-19. You should communicate this 5 day absenteeism to you faculty OR through the Student Life Excused Absence Process. Students can complete the Excused Absence here: https://www.mines.edu/student-life/forms/personal-excused-absence/.
- For additional support, please reach out to one of the many resources on campus:
- Academic support: Email CASA at firstname.lastname@example.org.
- Student Life Office: Excused absence(only if you must be away from class for more than three days).
- Experiencing a mental/emotional challenge? Email Student Outreach & Support (SOS) at email@example.com or fill out a SOS Referral.
The Mines default position is to trust students unless they give us a reason not to. Please think carefully before attempting to abuse this trust. As long as your number of short-term illnesses is low, we will not require anything beyond the answers to the questions below to mark you as excused, and you can work through what you missed for your own understanding. However, if you end up requesting a large number of excused absences, suspicious absences, or opportunistic absences, then we may follow-up and require you to submit proof that you have worked through the missed material before excusing you (in part, to make sure you are not falling behind). If we determine that you are violating the intent of this policy, then we will treat it as Academic Misconduct or a Code of Conduct matter.
Information to provide in email to instructor:
- Name and CWID
- Course Name
- Section and/or Time
- Class day missed
- Do you certify that you are indeed not feeling well and that after submitting this request will be limiting your activity to recover and/or lessen exposure to others as much as possible until you feel better?
Diversity and Inclusion
At Colorado School of Mines, we understand that a diverse and inclusive learning environment inspires creativity and innovation, which are essential to the engineering process. We also know that in order to address current and emerging national and global challenges, it is important to learn with and from people who have different backgrounds, thoughts, and experiences.
Our students represent every state in the nation and more than 90 countries around the world, and we continue to make progress in the areas of diversity and inclusion by providing Diversity and Inclusion programs and services to support these efforts.
Disability Support Services
The Colorado School of Mines is committed to ensuring the full participation of all students in its programs, including students with disabilities. If you anticipate or experience any barriers to learning in this course, please feel welcome to discuss your concerns with me. Students with disabilities may also wish to contact Disability Support Services (DSS) to discuss options to removing barriers in this course, including how to register and request official accommodations. Please visit their website at disabilities.mines.edu for contact and additional information. If you have already been approved for accommodations through DSS, please meet with me at your earliest convenience so we can discuss your needs in this course.
Accessibility within Canvas
Read the Accessibility Statement from Canvas to see how the learning management system at the Colorado School of Mines is committed to providing a system that is usable by everyone. The Canvas platform was built using the most modern HTML and CSS technologies, and is committed to W3C’s Web Accessibility Initiative and Section 508 guidelines.
Discrimination, Harassment, and Title IX
All learning opportunities at Mines, including this course, require an environment that allows each student to be able to learn without fear of discrimination or harassment based on any protected class. Mines’ core values of respect, diversity, compassion, and collaboration will be honored in this course, and the standards in this class are the same as those expected in any professional work environment. (More information can be found here.) Discrimination or harassment of any type, including sexual harassment, sexual assault, dating violence, domestic violence, and stalking, are prohibited under the Policy Prohibiting Unlawful Discrimination and the Policy Prohibiting Sexual Harassment, Sexual Assault, and Interpersonal Violence. As a participant in this course, we expect you to respect your instructor and your classmates. As your instructor, it is my responsibility to foster a learning environment that supports diversity of thoughts, perspectives and experiences, and honors your identities. To help accomplish this:
- Course rosters are provided to the instructor with the student’s legal name. I will honor your request to address you by a preferred name and I will use your identified pronouns. Please advise me of this preference early in the semester so that I may make appropriate changes to my records.
- If something is said or done in this course (by anyone, including myself) that made you or others feel uncomfortable, or if your performance in the course is being impacted by your experiences outside of the course, please report it to me (if you are comfortable doing so), to the https://www.mines.edu/institutional-equity-title-ix/reporting/, and to https://www.mines.edu/institutional-equity-title-ix/submit-report/ (an anonymous option).
In this course, we will cultivate a community that supports survivors, prevents interpersonal violence, and promotes a harassment free environment. Title IX and Colorado State law protects individuals from discrimination based on sex and gender in educational programs and activities. Mines takes this obligation seriously and is committed to providing a campus community free from gender and sex-based discrimination. Discrimination, including sexual harassment, sexual violence, dating violence, domestic violence, and stalking, is prohibited and will not be tolerated within the Mines campus community. If these issues have affected you or someone you know, you can access the appropriate resources here: http://www.mines.edu/title-ix/. You can also contact the Mines Title IX Coordinator, Carole Goddard, at 303.273.3260 or firstname.lastname@example.org for more information. It’s on us, all of the Mines community, to engineer a culture of respect.
CARE @ Mines
If you feel overwhelmed, anxious, depressed, distressed, mentally or physically unhealthy, or concerned about your wellbeing overall, there are resources both on- and off-campus available to you. If you need assistance, please ask for help form a trusted faculty or staff member, fellow student, Dean of Student Office (Raise Your Hand), or any of the resources below. As a community of care, we can help one another get through difficult times. If you need help, reach out. If you are concerned for another student, offer assistance and/or ask for help on their behalf. Students seeking resources for themselves or others should visit care.mines.edu.
Additional suggestions for referrals for support, depending on comfort level and needs include:
- CARE at Mines: care.mines.edu for various resources and options, or to submit an online “CARE report” about someone you’re concerned about, or email email@example.com
- CASA – https://www.mines.edu/casa/ for academic advising, tutoring, academic support, and academic workshops
- Counseling Center – https://www.mines.edu/counseling-center/ or students may call 303-273-3377 to make an appointment. There are also online resources for students on the website. Located in the Wellness Center 2nd floor. Located at 1770 Elm St.
- Health Center – https://www.mines.edu/student-health/ or students may call 303-273-3381 for appointment. Located in Wellness Center 1st floor.
- Colorado Crisis Services – For crisis support 24 hrs/7 days, either by phone, text, or in person, Colorado Crisis Services is a great confidential resource, available to anyone. http://coloradocrisisservices.org , 1-844-493-8255, or text “TALK” to 38255. Walk-in location addresses are posted on the website.
- Food and/or Housing – Any student who faces challenges securing their food or housing and believes this may affect their performance in the course is urged to contact the Dean of Students for support. Furthermore, please notify your professor if you are comfortable in doing so. This will enable your professor to provide resources that may be available.
All of these options are available for free for students. The Counseling Center, Health Center, and Colorado Crisis Services are confidential resources. The Counseling Center will also make referrals to off-campus counselors, if preferred.
In an emergency, you should call 911, and they will dispatch a Mines or Golden PD officer to assist.
Center for Academic Services and Advising (CASA)
CASA (https://www.mines.edu/casa) provides a variety of services to support students during their time at Mines.
- Academic Advising: All students are advised by Academic Advising Coordinators in CASA throughout their undergraduate studies at Mines.
- Academic Support Services: CASA offers a wide variety of support services designed to assist students throughout their undergraduate degree. Examples include pre-finals workshops, major exploration events, and the specific support services listed below.
- Tutoring: Tutoring services are offered for all core curriculum courses and many major courses by peers. Tutoring is offered Sunday through Thursday in CASA (Aspen Hall), the Library, and via zoom.
- Core Review Sessions: Core Review Sessions are group review sessions held by a peer facilitator before common core course exams. Peer facilitators also host regular office hours for more individualized assistance.
- Academic Coaching: Students can work with CASA Advisors to develop the skills and technique of studying well in college, such as test-prep and cognitive learning development, in a one-on-one setting.
- Faculty in CASA: Faculty from various departments host their regular office hours in CASA. Students are encouraged to utilize these professors for assistance with material and/or questions on course planning.
The Writing Center
The writing center is a free academic support service available to all members of the campus community including undergraduate and graduate students. We can assist you at any stage of the writing process, from brainstorming to final revisions. You do not need a complete draft to make an appointment. Our consultants are experts in a variety of composition and communication fields, providing support as you work on projects such as lab reports, essays, collaborative papers, scholarly publications, thesis chapters, and oral presentations. Whether you are focusing on organization or sentence structure, the Writing Center can evaluate your individual needs and tailor each appointment so that you become a more effective and efficient communicator. The Writing Center is open Sunday-Friday for in-person and online appointments. To learn more about our services and to make an appointment, please visit writing.mines.edu. For questions, please e-mail firstname.lastname@example.org.
Policy on Academic Integrity/Misconduct
The Colorado School of Mines affirms the principle that all individuals associated with the Mines academic community have a responsibility for establishing, maintaining an fostering an understanding and appreciation for academic integrity. In broad terms, this implies protecting the environment of mutual trust within which scholarly exchange occurs, supporting the ability of the faculty to fairly and effectively evaluate every student’s academic achievements, and giving credence to the university’s educational mission, its scholarly objectives and the substance of the degrees it awards. The protection of academic integrity requires there to be clear and consistent standards, as well as confrontation and sanctions when individuals violate those standards. The Colorado School of Mines desires an environment free of any and all forms of academic misconduct and expects students to act with integrity at all times.
Academic misconduct is the intentional act of fraud, in which an individual seeks to claim credit for the work and efforts of another without authorization or uses unauthorized materials or fabricated information in any academic exercise. Student Academic Misconduct arises when a student violates the principle of academic integrity. Such behavior erodes mutual trust, distorts the fair evaluation of academic achievements, violates the ethical code of behavior upon which education and scholarship rest, and undermines the credibility of the university. Because of the serious institutional and individual ramifications, student misconduct arising from violations of academic integrity is not tolerated at Mines. If a student is found to have engaged in such misconduct sanctions such as change of a grade, loss of institutional privileges, or academic suspension or dismissal may be imposed.
The complete policy can be found in the Mines’ Policy Library.
If you face uncontrollable, unusual, or extreme challenges and are able to provide appropriate documentation, I am happy to make reasonable accommodations for you to make up a missed quiz, lab, or homework assignment.
Extra credit may be offered for additional learning activities related to this class. Unless otherwise detailed in this syllabus, the awarding of extra credit is at the discretion of the instructor and is not guaranteed.
Coursework Return Policy
The course staff will try its best to grade and return assignments to you within 2 weeks of submission, along with suitable materials/feedback that enable students to understand how to improve their learning/performance.
Course Issues and Concerns
As part of good professional practice, students are encouraged to speak with the faculty directly to raise issues and concerns with regards to the course professionally in compliance with the student code of conduct. Students can also reach out to the course coordinator Dr. Tom Williams or the head of the department of Computer Science Dr. Iris Bahar the course is being offered through. The department head can investigate and work with the faculty member to resolve course-related concerns. Students’ final point of contact is the Dean of Energy and Materials Programs, Dr. John Berger, who can make any final decisions.
Participating in this Course
Expectations for Participation
You are expected to engage in all course activities, tasks, and assignment as an emerging professional.
You are expected to spend between TBD-TBD hours on this course each week during out-of-class time, in addition to actively participating during the class-time each week.
Profile in Canvas
As part of the learning experience at the Colorado School of Mines, our class will be utilizing online learning resources and experiences through the Canvas learning management system. In order to help build community in this online learning environment, you are encouraged to upload your profile picture to Canvas. Photos should be similar to the photos taken for passports or state identification cards.
Expectations of online etiquette or netiquette
Here are few do’s and don’ts about communicating in your course through emails or in online discussion forums:
- Ask questions and engage in conversations as often as possible—feel free to contact the instructor via the discussion forum for questions or via email or other communication.
- Be patient and respectful of others and their ideas and opinions they post online.
- Remember to be thoughtful and use professional language. Keep in mind that things often come across differently in written text, so review your writing before posting.
- Be prepared for some delays in response time, as “virtual” communication tends to be slower than “face-to-face” communication.
- Contact the instructor if you feel that inappropriate content or behavior has occurred as part of the course.
- Check the syllabus and course policies stated by your instructor to know what to expect about your instructor’s turnaround time for responding.
- Do NOT…
- Use inappropriate language—this includes, but is not limited to, the use of curse words, swearing, or language that is derogatory.
- Post inappropriate materials—for example, accidentally posting/showing a picture that is not appropriate for the course content.
- Post in ALL CAPS, as this is perceived as shouting and avoid abbreviations and informal language (“I’ll C U L8R”).
- Send heated messages even if you are provoked. Likewise, if you should happen to receive a heated message, do not respond to it.
- Send an email or post to the entire class, unless you feel that everyone must read it.
Note that we use Canvas for
- course content (Modules),
- submissions (Assignments),
- grading (Grades),
- and communication (Discussions).
|1||8/23 (T)||Course Intro|
Intro to AR
|Chapter 1 (free sample chapter)||HW1: Administrative stuff (Due 8/25)|
HW2: AR killer apps (Due 8/25)
Dr. Zhao Han’s Research on AR
|1||8/25 (Th)||Unity: Intro||HW3: Roll-a-Ball (Due 8/30)||HW1, HW2|
|2||8/30 (T)||AR Displays|
|Chapter 2||HW4: “Use cases” for displays (Due 9/1)||HW3|
|2 (Sep)||9/1 (Th)||Vuforia: Intro||USB webcam (return 12/1)|
Printed image target
HW5: Clock animation (Due 9/6)
A1: Augmenting Real Objects (Due 9/20)
|HW4 (peer review due next day)|
|3||9/6 (T)||Image Targets|
Vuforia: Image Targets
|Chapter 3 (3.1, 3.2, 3.6)|
Chapter 4 (4.1, 4.3, skim algorithms)
|HW6: Vuforia image target (Due 9/8)||HW5|
|Chapter 4 (4.4-4.6, skim algorithms)||HW6|
|4||9/13 (T)||Non-visual Tracking and Sensor Fusion|
|Chapter 3 (3.3-3.5, 3.7)||HW7: Animation (Due 9/15)|
|4||9/15 (Th)||Visual Coherence||Chapter 6||HW7|
|5||9/20 (T)||Unity: Lighting, Shadows|
Assigning Magic Leap 1 devices to people who need one
|Magic Leap 1 devices|
HW8: Commercial app visual coherence (Due 9/22)
A2: Visual Coherence (Due 9/29)
|5||9/22 (Th)||Situated Visualization||Chapter 7||HW9: Hyper-Reality (Due 9/27)|
Technical Topic Proposal (Due 10/6)
|6||9/27 (T)||Magic Leap 1 Deployment|
Android and iOS Deployment
|6||9/29 (Th)||Interaction 1||Chapter 8 (8.1-8.2)||HW10: Deployment|
Final Project Proposal (Due 10/13)
|7 (Oct)||10/4 (T)||Interaction 2|
|Chapter 8||HW11: Select an object (Due 10/11)||HW10|
|7||10/6 (Th)||Unity: Physics|
|Technical Topic Presentation (Due 10/27-11/8)||Technical Topic Proposal|
|8||10/11 (T)||Vuforia: Virtual Buttons||A3: Interaction (Due 10/25)|
HW12: App brainstorm (Due 10/20)
|8||10/13 (Th)||Discuss app ideas from brainstorm session||Final Project Proposal|
|9||10/18 (T)||No Class (Fall Break)|
|9||10/20 (Th)||Plane Finding|
|10||10/25 (T)||AR Foundation|
|Schedule of presentations||A3|
|10||10/27 (Th)||Technical Topic Presentation||Schedule of presentations|
|11 (Nov)||11/1 (T)||Technical Topic Presentation||Schedule of presentations|
|11||11/3 (Th)||Technical Topic Presentation||Schedule of presentations|
|12||11/8 (T)||Technical Topic Presentation||HW13: Guest Speaker (Due 11/10)|
|12||11/10 (Th)||Guest Speaker (Dr. Tom Williams on AR in Human-Robot Interaction, tentative)||HW14: Guest speakers (Due 11/15)||HW13|
|13||11/15 (T)||Guest speaker|
|13||11/17 (Th)||No lecture today, TA in-class help with projects||Final Project Teaser (Due 11/22)||HW14|
|14||11/22 (T)||Final Project Teaser Presentations||Final Project Teaser|
|14||11/24 (Th)||No Class (Thanksgiving Break)|
|15||11/29 (T)||Final Project Teaser Presentations|
|15 (Dec)||12/1 (Th)||Final Project Demo||Final Project Reviews (Due 12/6)||Webcam/Magic Leap 1 devices return before leaving|
|12/6 (T)||Final Project Reviews|
This course was approved on April 20, 2022.