By Lukas Timm · 6 October 2026 · Experiment ongoing
EP03 · 1:06 · Recorded evidenceUsing a checkerboard displayed on a laptop as a calibration target in the real room. Watch this moment ↗
I paired a roughly $5 Raspberry Pi camera with an Ouster OS0 LiDAR and asked GPT-6 to help make them agree. We got the room edges much closer. Then I sat in the scene and found a ghost of myself on the wall.
Episode 3 is published. Watch the episode ↗. This field note preserves the 6 October experiment snapshot; the investigation continued in later episodes.
First, I was the calibration stand.
The beginning was wonderfully unglamorous: walking around with a checkerboard on my MacBook, collecting twenty images, then clicking corresponding features in the camera and LiDAR views. The checkerboard helped measure the camera’s lens. Matching the two sensors still needed work.
The agent could write the geometry, extract edges, run fits and compare results. I kept challenging the matches: that door edge, that shelf, that same physical boundary in both views. A lower score was only useful if it described the right thing.
EP03 · 2:34 · Recorded evidenceReusing the same physical room boundaries to check the projection. Watch this moment ↗EP03 · 1:28 · Recorded evidenceManually matching a room feature between the camera image and LiDAR view. Watch this moment ↗
Real progress. A smaller claim.
In an earlier static comparison, the mean of seven selected boundary median errors fell from about 13.5 to 4.2 pixels—roughly a 69% reduction. That was an improvement on those room features, not a measurement of overall system accuracy or proof that every point would receive the right color.
EP03 · 2:56 · Recorded evidenceThe selected static-boundary comparison improves from 13.5 to 4.2 pixels; this is not whole-system accuracy. Watch this moment ↗
Then the room acquired a second me.
When I entered the scene, foreground colors appeared on the wall. The first filtering attempts did not solve it. The ghost remained, and parts of the point cloud disappeared.
EP03 · 3:40 · Recorded evidenceAdding a person reveals incorrect colour assignment in the point cloud, despite improved room alignment. Watch this moment ↗
That was not a result I wanted to hide. A cleaner-looking image is not enough if the system gets there by deleting parts of the measured scene.
Four poses. One rocking horse.
We collected a seated reference, an extended arm, a raised toy and a forward lean. I tried to hold still. These captures let us ask whether a correction that helped the room also helped nearby objects.
EP03 · 4:02 · Recorded evidenceFour held poses used to investigate the remaining camera–LiDAR mismatch. Watch this moment ↗
More fitting brought mixed results. Some outlines improved while others got worse. We tried a new camera pose in the live view; the wall artifact was still there. The experiment needed better diagnosis, not another victory screenshot.
Three questions, not one magic adjustment.
Alignment: does the same physical feature project to the right pixel?
Timing: did the camera and LiDAR observe the same moment?
Visibility: does that camera pixel actually show the surface we are coloring?
A wall point can be real while the pixel copied onto it shows my shirt. Visibility handling is a separate part of texture mapping; PCL’s texture-mapping documentation includes explicit occlusion checks. That does not prove visibility is the only cause here: our physical edges still show offsets, and arrival times do not tell us the exact exposure timing.
The next check has to survive the room.
The audit now separates projection code, timing, independent calibration and visibility-aware color assignment. The browser and OpenCV projection implementations agreed across 500 test points. That checks the code’s consistency; it does not validate the physical calibration.
Timestamp pairing is now working, using an estimated relationship between the sensor clocks. That is not hardware synchronization. A newer surface-based visibility trial retains every measured point and leaves uncertain colors unassigned. Some duplicated color is removed, but spill remains around the body and toy. An independent calibration baseline still needs to be run.
The target is simple to describe: keep my colors on me, retain the measured wall, and leave a color unknown when the camera cannot support it. A verified fix is still outstanding at this snapshot.
That is the useful part of building with an agent for me. It can carry out much of the mathematics and programming. I can spend more attention on the experiment—and keep asking whether the result matches what is actually in front of us.
Evidence snapshot: 6 October 2026, through the projection audit, timestamp-pairing update and incomplete surface-visibility trial. Images on this page are frames from the published episode. Text prepared with AI assistance and grounded in the experiment logs. The experiment was unfinished at this snapshot; the episode is now published. Read how the sensor work developed across the series →.
