AI in Motion

Author(s): michael raspuzzi

Originally published on Towards AI.

AI in Motion

24-hour hardware hack recap in San Francisco

If 2023 was the year of LLMs (large language models), then 2024 will be the year of LMMs (large multimodal models). The main difference will be the recognition of text and images for generating inputs and outputs. This opens up a whole new set of possibilities for hardware.

To explore what’s possible in combining the latest hardware with the latest machine learning models, we hosted a weekend-long hackathon at Studio 45 in San Francisco. The main intention was to bring together two communities that are rather separate right now: the machine learning community and the robotics community. Hackers came together to see what spatial challenges they could solve. Everyone had 24 hours to make a team, build a demo, and pitch judges at the end.

Smarter interfaces, GPT-4 with eyes, and an open source model from deepmind

In the two weeks around the hackathon, there are new developments in the realm of hardware and AI:

• Investment in smarter interfaces: Meta released Ray-Ban glasses with a streaming setup. OpenAI is in talks with Jony Ive for an iPhone-like replacement using their latest models. Prior in the year, Humane demoed their AI pin on a TED stage, and Apple Vision Pro headset has released its SDK with visionOS as people build spatial applications.
• OpenAI gives Chat GPT-4 vision: Chat GPT-4V (vision) is now on premium accounts letting people chat with images as well as Dalle-3 for generated content. Some use cases include taking a front-end mockup and writing back-end code, making movie stills, or even figuring out confusing street signs. More examples here and Microsoft’s full 166-page paper on GPT-4V here.
• Google DeepMind open sources RT-X: Benchmarking over 500 skills on over 150,000 tasks, the RT-X model out-performed traditional narrow intelligence models, with more here.

While we missed these exciting updates by two weeks, this is exactly the reason behind running the AI in Motion hackathon in San Francisco: 1) the pace of machine learning advancement is happening so quickly that we want to see what happens when integrated better with hardware and 2) we wanted to bring two communities together that don’t often get to hack together.

Our main goal was bringing together ai + hardware

To structure the hackathon, we first started with three goals:

Goal 1: Ensure the group is ½ machine learning developers and ½ hardware engineers.

We achieved: Hackers from OpenAI, DeepMind, Meta AI Labs, Tesla, and more came with experience in both backgrounds.

Goal 2: Bring in some awesome hardware to see what the latest machine learning models could do. Yes, there were more than LLMs used in the making of these demos.

We achieved: We had universal robotic arms, a Boston Dynamics Spot quadruped, Roombas, and a whole IoT kit library. Here’s the full documentation we gave hackers. Look out for an open source document soon!

Goal 3: Go from idea to demo in 24 hours. We launched Saturday at 10 a.m. and demos were finalized by 10 a.m. on Sunday. Yes, many still got to sleep.

We achieved: Check out the results below!

Overall, the results were impressive. It felt like a taste of where the very near future will be drastically different with smarter interfaces, more capable hardware, and more importantly, smaller teams shipping bigger builds.

Top 5 project highlights

1. Jarvis is a robo-mechanic assistant. Think Tesla manufacturing arm, but in a small-scale garage, and one you can talk to.

h/t to the hackers: @jqphu, @nishthenomad, @TristanHeywood, @The_TT_Hacker , @winston, @vrushank

2. XR is a smart learning hearing aid that can capture surrounding information, like who’s around the user and what’s being done. Combining camera input with computer vision and voice recognition with OpenAI’s API, their demo could enhance a user’s visual understanding of the world around them.

h/t to the hackers: @jer, @EmmaQian_ , @ClovisVinant, @lingxue, @varun, @esh

3. C.H.I.P. is a digital CNC microscope with zero-shot classification for bad-chip detection.

h/t to the hackers: @johndmcmaster, @notionsmith, @justin, @ninjaa

4. Dex (overall winner) scans and queries a room for lost objects. They added a webcam to a Roomba and made it so you could chat with the image data.

h/t to the hackers: @cyrus_cowley, @ian, @surya

5. Spotsight (the crowd favorite) was a robotic seeing-eye dog for the visually impaired. It could both help navigate an environment safely, at a more affordable price than professionally trained seeing-eye dogs, as well as do additional tasks between owner and environment, like get the mail.

h/t to the hackers: @ingarobotics, @cyb3rblaze_, @adit, @reuben, @abinaya

U+1F916 More projects here if you’re curious.

Ideas to improve on the next one

• From idea to storyboard. We discovered a gap from teams finding a clear direction in what problem they generally wanted to solve and how they would demo that in 24 hours. For future hacks, we’ll promote two parts of the storyboard: 1) what you want the robot to do and why and 2) what your back-end architecture will look like. We course-corrected on the spot and can preemptively help with this sooner next time.
• Multi-part hack. The most challenging hackathons at MIT can take two weekends instead of one sitting. as it can take a full weekend to explore an idea and figure out what the demo will be, with a second weekend for the finalists to take time to build. We may explore this for the next challenge to see if it helps promote higher-quality demos. In this case, we’d keep the first weekend to one day, and the second weekend would be two days.
• Simplify the hardware library. We’re going to explore the two-weekend model with a more scoped-down library. It was a good problem to have too much hardware. But with each robot having a mentor, we think it may be better to do a scoped-down challenge like one with the Boston Dynamics Spot quadruped or the universal robotic arms. If we did this, we’d have to explore timing for teams to use the hardware or get multiple bots.

Use this as a playbook, if you want to run your own hardware hackathon

Our team has run dozens of hackathons, and we’re finding there are not enough hardware hacks. While it’s easier to ship some software, we hope this acts as inspiration and structure for future hardware hacks.

Some things we were intentional about that worked well:

1. use a design fabrication space that hackers can build in.

Shiny polished office desks are not recommended when soldering or tinkering with hardware. We also rented plastic tables to ensure most flexibility.

2. allow hackers to apply as a team.

2/9 teams applied as a team. They could do so in the application ensuring we accepted their whole team to the event. Feel free to use this page as a template for your own hack. We welcomed people of all backgrounds. The only requirement was you were there to build + ship something. no lurkers.

3. release a library of parts 1 week before the hack.

We shared this document 1 week prior to the event to let participants explore documentation. They were also encouraged to bring their own hardware.

4. kickoff strong.

We started the hack with coffee and informal mix + mingling. This enabled thirty minutes of people getting to know each other. then we had 45min of programming with a founder demo, Gary of raise robotics, Where he could share a real example of what to build for. Each robot also had a robot mentor who gave an introduction to the hardware.

5. for those who need a team, we helped them make one.

We did 1min stand up pitches, and 20min of mix and mingle three times on repeat until people met each other, formed inklings of teams, and then finalized teams.

6. be clear in the judging criteria.

We were clear since kickoff that the judges would be evaluating ideas based on:

• 6.1 Understand the problem: Is this a big problem or does this enable an interesting future? Is there a clear customer use case?
• 6.2 Demo: Is this relevant to solve the problem? — feasibility — does it prove the concept?
• 6.3 Playfulness: Is this fun or delightful to interact with? will this demo, and system in mass deployment, create joy using ai systems and motion?
• 6.4. Clear economic incentives: Is there a viable path to deploy at scale? clear reason customer would pay to use this?

7. submit ideas before robot pairing happens.

The next 2–3 hours after team formation, was idea generation. We intentionally separated this from building to ensure teams had an idea of what they wanted to build. We also had a meritocracy where the best ideas quickest to submit got first pick at their robots.

8. have mentors on both sides floating.

We had mentors on the design fabrication side, tooling, hardware, and machine learning side available to help teams. a team of 3–5 mentors helped all 50 hackers.

9. end strong.

Alongside submitting pitch decks (2min) and demos (1min), we wanted ot make sure there was a live element to the showcasehardware hack finale. We split the event into two parts:

• 1 hour of showcase: think startup booths with live demos. guests and judges could meet the teams and interact with the demos.
• 1 hour of pitches: 2min pitch and 1min video demo. the embedded youtube video in google slide demo ensured that everyone could see everything that was made.

after each pitch, each judge had the chance to ask 1 question or give 1 point of feedback.

10. structurally unstrusctured.

The majority of the hackathon was focus time where each team could take time to figure it out. We skipped the extra talks and workshops, to ensure teams had as much time as they needed. even the challenge was intentionally open ended: “solve a spatial challenge at home, in the office, or outside where ai pairs up with hardware.”

Wrapping up with movie credit style thank yous

This was a fun event both for hackers and for those who came for the final showcase. We need more hardware hackathons!

To build great hardware, it really does take a full ecosystem, even if only for 24 hours. and we’re grateful to all our sponsors who helped make this event happen, especially informal, which played a pivotal role!

If you want to run your own hardware hackathon, in a future post, we’ll be sharing tactical tips on how to run a hardware hackathon. Stay tuned!

Thank you to all the partners who helped make this happen!

• ​​Studio45 is a clubhouse and coworking space for professionals building physical products in the Bernal Heights neighborhood of San Francisco.
• ​informal is a freelance collective for the best independent professionals in hardware and manufacturing. informal members work with companies at every scale to design, manufacture, and ship physical products. Thanks for the feature!
• ​​Blues Wireless makes cloud-connected products actually possible with out-of-the-box connectivity.

​Community partners

• ​Massmelt is a diverse set of hardware product-development services dedicated to supporting organizations as they build their vision of the future.
• ​​SF Hardware Meetup is a community of 9,000+ hardware professionals meeting monthly to build meaningful connections.
• Cerebral Valley and GenAI Collective help spread the word to the Bay Area machine-learning community.

​Prize sponsors

• ​Runpod is a GPU cloud platform for training and scaling inference on AI models. They offered cloud credits to the top team.

Thank you to our judges for helping comment on the projects!

• ​Ashley is former tech lead at Google Alphabet’s X, Moonshot Factory.
• ​Robert is a product lead at Waymo, a self-driving company within Alphabet.
• ​Santhi is founder, angel investor, and venture fellow at Designer Fund.
• ​Vince is a 7x entrepreneur and Silicon Valley impact tech investor.

And last, but not least, shout out to the co-organizers: Michael + Jascha, both members of informal, for volunteering to make this event a great experience for everyone who joined..

P.S. Yes, this article was written by us (Michael + Jascha), and so we did end with a third person shout out at the end U+270C️

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI