Woah this is awesome. I imagine this could be used to make “shitty robots” with imperfect joints, backlash or other things more precise. Which could help with 3D printers as well as 3D printed robots. Or using internal strings and pulleys to control a robot arms. Especially for larger robot arms stiffness becomes a problem.
Is there an open source, easy to use software framework so people can start to play and experiment with it?
They really went all out haha. They also have a linked website with a video and a tutorial. The tutorial has an easier to understand explanation of what they are doing.
It seems they use simple motion flow of the video to train the neural net, but they also use some kind of volume rendering to train the AI to predict and reconstruct the 3D scene with your robot. And they use cheap depth cameras but apparently it also works without depth. And this works for basically any robot you can imagine which is really brilliant.
Looking at all those pneumatic soft robots, now I wonder if you could invert this to use for an 3D input device. Like a kind of 3D printed pneumatic joystick that simply measures the resulting air pressure at the end of internal channels when you tilt or twist or move the joystick. No wiring or assembly, just 3D print a joystick and glue it to a board.
I suspect it might even be easier (forward vs inverse kinematics).
If you could combine both (pressure sensors and pistons) you could do force feedback input devices that can be 3D printed. Which could be useful for serious applications, like controlling a vehicle and “feeling” the road, or feeling the air pressure on the wings of a plane. Or controlling an excavator and “feeling” the earth by measuring feedback in the pneumatic pressure. Currently things like that is really expensive.
MCasq_qsaCJ_234@lemmy.zip
on 19 Jul 22:12
nextcollapse
threaded - newest
Woah this is awesome. I imagine this could be used to make “shitty robots” with imperfect joints, backlash or other things more precise. Which could help with 3D printers as well as 3D printed robots. Or using internal strings and pulleys to control a robot arms. Especially for larger robot arms stiffness becomes a problem.
Is there an open source, easy to use software framework so people can start to play and experiment with it?
yup, here’s the repo for it github.com/sizhe-li/neural-jacobian-field
They really went all out haha. They also have a linked website with a video and a tutorial. The tutorial has an easier to understand explanation of what they are doing.
It seems they use simple motion flow of the video to train the neural net, but they also use some kind of volume rendering to train the AI to predict and reconstruct the 3D scene with your robot. And they use cheap depth cameras but apparently it also works without depth. And this works for basically any robot you can imagine which is really brilliant.
Looking at all those pneumatic soft robots, now I wonder if you could invert this to use for an 3D input device. Like a kind of 3D printed pneumatic joystick that simply measures the resulting air pressure at the end of internal channels when you tilt or twist or move the joystick. No wiring or assembly, just 3D print a joystick and glue it to a board.
Using the system in reverse seems like an interesting idea. I can’t see why that wouldn’t work either.
I suspect it might even be easier (forward vs inverse kinematics).
If you could combine both (pressure sensors and pistons) you could do force feedback input devices that can be 3D printed. Which could be useful for serious applications, like controlling a vehicle and “feeling” the road, or feeling the air pressure on the wings of a plane. Or controlling an excavator and “feeling” the earth by measuring feedback in the pneumatic pressure. Currently things like that is really expensive.
Take note of that, John Carmack, and improve it.
A headline that says nothing at all, lol.
When you’re commenting on a subject you have no clue about lol.