Also, I can't find "maximum number faces", so the update seems not to be reflected properly. Is there anything I need to do to use the updated version?
I apologize; mistake is made at my end :tear: . I forgot to rename the new web file as "index". In effect, itch.io read the old web file and ignored the new. I have corrected the name and should reflect the updated interface. Good thing you mentioned it :handshake: .
Sorry for the late response, thank you for reflecting the updated interface! I have tried to do the multiple face tracking, and the following video is the result (Sorry for the video looks a bit confusing because the position of the model and the face being tracked are reversed):
It worked at first, but it stopped after about 20 seconds :think:
The tracking itself seemed accurate while it was working, so I’m very impressed when I saw it for the first time! I don't know the cause why it would stop, but it would be a very interesting feature if it could be made to work consistently.
I haven't tried the other settings yet, so I'll try them later!
I have not managed to break the multiple face tracking and replicate the bug in your video. This is something that would require further investigating. I would like to know if you could replicate this bug repeatedly. It could be either hardware or software that is causing the bug. For one, the camera only allow one application to read the web camera stream data from what I have tried on Windows. Since the web application does not write any log files onto your computer or communicate to a server, it will be harder to pinpoint the cause of the bug.
I would like to know if you could replicate this bug repeatedly.
Yes, I was able to replicate this bug again. I think it's possible that I have the wrong settings, so I recorded the process from when I accessed the tool:
One thing I noticed is that the model that was initially tracking my movement will follow the second face’s movement when the second face tracking starts. Is this something wrong with my settings?
Can you open up the web inspector ( should be F12 shortcut )? See if there are any errors in the console section.
Misaki wroteOne thing I noticed is that the model that was initially tracking my movement will follow the second face’s movement when the second face tracking starts. Is this something wrong with my settings?
It is not your settings. The face tracking system is probably scanning the camera image from the top to down and left to right direction. The second face that appeared after is identified as face 1 because it is top left of the first face.I am going to add an numeral indicator near face meshes later to help identify the face index.
The face tracking system is probably scanning the camera image from the top to down and left to right direction. The second face that appeared after is identified as face 1 because it is top left of the first face.I am going to add an numeral indicator near face meshes later to help identify the face index.Ah, I see. I am relieved to hear that my settings were not wrong.
I tried again and found an error in the console tab.
Uncaught (in promise) TypeError: Cannot read properties spine_vtuber.js:1 of undefined (reading 'x')
at V (spine_vtuber.js:1:13566)
at l (spine_vtuber.js:1:13819)
at face_mesh.js:73:321
Here is a video recorded with the console tab open.
I think I have located the problem from the error message and fixed it. Try again and see if you still get the error. For some reason your web browser was generating incomplete data. I made the program check if the data is missing beforehand so it would not stop working :wounded: .
Thank you for your quick response! Unfortunately, I still got the error again. I didn't open the error log details last time so I showed that at the end of this video:
Let me know if I need to try anything else!
The good news is that the previous fix worked but the same problem occurred on another section. I did the same check up on that section as well. I hope I squashed all those bugs :wounded: .
I am surprised that those errors has not appeared on my computer with multiple face tracking :think: .
Now it's stable and working! Awesome 
However, as you can see in the video, only the model placed on the right side sometimes momentarily appears to be in a strange look where each facial part seems to pop outward.
I checked the recorded video frame-by-frame and it seems that sometimes the face goes off the tracking and when it comes back on, the face parts seem to be pulled as shown above.
I don't know why the tracking is off, especially since the face is not moving significantly, but the face closer to the camera appears stable, so it may not be stable if it is farther away.
Now it's stable and working! Awesome
Great to hear.
However, as you can see in the video, only the model placed on the right side sometimes momentarily appears to be in a strange look where each facial part seems to pop outward.
I had the animation track scaling factor too high for when the AI stopped recognizing a face. The initial value of the vector has been adjusted by a million fold. Hopefully that will not produce an image that is as jarring as your image. I added another circumvention to stop moving the Spine model in that situation as well.
I don't know why the tracking is off, especially since the face is not moving significantly, but the face closer to the camera appears stable, so it may not be stable if it is farther away.
I lowered the face recognition threshold from 80% to 50%. When the AI is at least 50% confident, the computer will start producing facial coordinates for the Spine model. Hopefully that will help with finding faces further away from the camera. I am not sure if I want to expose the ability to change the threshold in the future.
The Spine Vtuber Prototype 1.0.4 works fine! This time, I did not see any errors that I saw yesterday. I also feel that faces can be detected well at a distance.
I also tried changing the strength settings. Some of the settings could be changed to get better effects (e.g. "face pitch" and "face yaw"), but it was difficult to get good results for the eyes because it sometimes did not work so well depending on the angle of the face. For example, turning the face down can cause the eyes to open wide.
As for the mouth, my skeleton setup is not good (it opens up against the top, which is not appropriate) and I would like to fix it when I have time.
Anyway, it is wonderful that this tool has improved very much in this short period of time :yes:
I think the eye issue could be resolved by editing the animation in Spine.
Anyway, it is wonderful that this tool has improved very much in this short period of time :yes:
Good thing they were simple fixes :wounded:.
This is really cool! I haven't tried anything VTuber related until now so this was a fun experience for me. :grinteeth:
Dev stuff isn't my area of expertise so I'm not exactly sure what you can do, but here are all the thoughts I had while I was using it:
Overall really cool to use and I'm excited to see how you take this further!
- It would be cool to see what files are added to the drag and drop zone when they're dropped in to know that they've successfully been uploaded without pressing the button below to see if it works.
Yes I can do something about that.
- I'm not sure what happened, but when I imported my latest rig, the import wouldn't load all of my assets. It worked when I used an older file though, so it may be an issue with my file and not the prototype (the same thing happened recently when I used that file with Rhubarb as well). But I thought I'd at least mention it (see bad import jpg)
I do not know either about the bad import. Would be hard to figure without any of the files or error log.
- Being able to adjust the strength of individual parameters was really helpful here. Before I recorded, my eyelids wouldn't open to their full open position so I adjusted that a bit
Yeah, not everyone is comfortable or able to stretch their facial features to the extreme. Even if they could, their face would fatigue eventually.
- I made a rough test of some eye blinks which I show in the video. I experimented with turning alpha to 0 on the lower lids so they would disappear when the eyes opened. It would be cool if there was a way to disappear the lids just as they open all the way.
- It would be even cooler if there was a way to use an entire animation as the motion rather than a single key (unless I set mine up incorrectly). For example, having one animation control the entire left eye blink. The first frame would be eye completely closed, and the last frame would be the eye completely open. Then the frames in between could be refined to allow custom deformation rather than a linear straight shot from one position to the next. This could also help in selecting the best time to adjust draw order too. I'm thinking of the Moho rigging process as inspiration here.
The problem is that the live timeline ( from face tracking ) would conflict with the animation timeline using animation mix alpha setup I have now. I think it is still possible but it would require the calculations to be applied to the animation track timeline itself and have the animation mix alpha constant at one. This would require me to create a separate copy of the application to experiment with. Wait for the next update. :nerd:
I tested with only with one eye but your request of able to use more than one keyframe for Spine Vtuber Prototype is possible! I keyframed at half way mark on the timeline with green eye and the full way with red eye. I had set the FPS to 100 so each frame is 1% of the movement range. You can set the FPS to any value but you have to keyframe within the one second. I still need to make the changes for the rest of the animation tracks and update it on itch.io :cooldoge:.
Spine Vtuber Prototype 1.0.5 update
https://silverstraw.itch.io/spine-vtuber-prototype
I have also updated the Spine Vtube Test Model to reflect the change in 1.0.5
The new animation system looks really interesting! However, I have tried the v1.0.5 with my skeleton which has not been changed since the previous test, but it did not work. It doesn't work when I turn on the camera active, and it has been in a weird state from the start. Here is the screenshot:
spine_vtuber.js:1 Uncaught TypeError: Cannot read properties of undefined (reading 'length')
at l (spine_vtuber.js:1:12224)
at t.ondrop (spine_vtuber.js:1:12560)Does the error log above seem to indicate the cause?
I was able to recreate your error and patched it. I forgot that drag and drop uses a slightly different code than clicking to upload files.
The weird state of your model is normal and is separate from the error. Before you were only allowed to keyframe the end state of the animation movement. Now that you can key more than one frame, you are going to have move the previous keyframe to the where 1 second would be at. For example. if your Spine file is set at 60 FPS, 1 second is on the 60th frame. If it set to 30 FPS, then 30th frame is the 1 second mark. Then you are going to need to add a keyframe at the 0th frame. It is going to take some time to make the changes to the Spine file but you are free to keyframe whatever you want within that 1 second timeline. Note: I set my FPS to 100 so that each frame is 1% of the movement range.
Thank you for your quick response! I understand the new specification of the animations. I fixed my skeleton and now it is working very well!
It seems to be smoother than before, is this also thanks to the update? Also, when testing multiple face tracking, there was a problem that if you opened the mouth as wide as possible, the mouth would open wider than specified in the "mouth height" animation, but this problem has been fixed and the model no longer moves wider than the created animation. This is really awesome.
Excellent. You got your model to work now.
I also noticed that your model animation got smoother from the video. I think the update definitely played a role. As to why, I would not be able to provide a concrete answer. I speculate it could better in-betweening when there is more than one keyframe. The other thing could be it better performance changing the track time as opposed to changing animation track alpha. Nate would probably know more about it.
The " the mouth would open wider than specified in the 'mouth height' animation" is a number capping issue. Out of the box, Spine runtime caps off the track time with track ending time while the animation track alpha is uncapped.
Thank you for explaining about them! It seems like a lot of things worked better by using the track time instead of using track alpha to change animations, right? That’s an interesting finding.
By the way, this change has also made the creation of the model much easier. It used to be difficult to make sure that the eye's mesh was clean when in the middle of the eye-open animation, but now it is easy to check and modify.
I think the animation of turning the face sideways could be improved, and I would like to modify it eventually.
Thank you for explaining about them! It seems like a lot of things worked better by using the track time instead of using track alpha to change animations, right? That’s an interesting finding.
The benefit of using the track time is much noticeable on your character model than my test model. Weighted meshes behave better when they are animated on a timeline as opposed to being a single keyframe.
I started using the track alpha because I saw it in a demonstration code and mimicked the usage. I had to research the Spine runtime documentation to find information about track time.
By the way, this change has also made the creation of the model much easier. It used to be difficult to make sure that the eye's mesh was clean when in the middle of the eye-open animation, but now it is easy to check and modify.
Awesome! I always wanted the creation process to be easy.
I think the animation of turning the face sideways could be improved, and I would like to modify it eventually.
Are speaking about your character model or the animation tracks setup in the Spine Vtube Prototype?
Are speaking about your character model or the animation tracks setup in the Spine Vtube Prototype?
Oops, sorry for not being obvious about it. I meant to talk about improvements to my model. My current model has just a narrow angle that can be moved left or right, but thanks to the ability to move the character using the track time, I expect that I should be able to switch the attachment at a specific time and have it face completely sideways. I'm hoping to get that done at some point!
My current model has just a narrow angle that can be moved left or right, but thanks to the ability to move the character using the track time, I expect that I should be able to switch the attachment at a specific time and have it face completely sideways. I'm hoping to get that done at some point!
I am not sure if the face tracking can turn all the way to the side and behave correctly. You can increase the strength of the face yawing signal so you do not have to turn your head completely sideways.
For the next update, I am working on adding index numbers to the face mesh to help sort the confusion with multiple face tracking and corresponding Spine models. I also want to work on saving and loading JSON files for Prototype settings.
I am not sure if the face tracking can turn all the way to the side and behave correctly.
Probably it is no problem, because what I'm planning is that I'd like to make it so that the character moves larger, such as turning sideways about 60 degrees, while the real face only moves at about 40 degrees.
For the next update, I am working on adding index numbers to the face mesh to help sort the confusion with multiple face tracking and corresponding Spine models. I also want to work on saving and loading JSON files for Prototype settings.
Those things would be very useful! I'm looking forward to the next update
I have a working saving and loading mechanism added to the project. The downloadable JSON will have a ( .svp ) extension name to avoid confusing with the Spine exported JSON. I will update https://silverstraw.itch.io/spine-vtuber-prototype in a few days. Hopefully there are not any bugs to fix so I can start working on expanding additional animations and skins implementation.
1.0.6
I have updated my skeleton and deleted the old one.
chara-for-Spine-Vtuber-Prototype.zip
@SilverStraw I'm surprised the v1.0.6 release came sooner than I thought! I confirmed that it works fine, but personally I wanted to save the skin setting, so it was a bit sad that the skin is not included in what is saved in the .svp file. (In my case, my skeleton has green and blue eye skins, but it's a bit of a hassle to have to set the skin each time I load it.)
Misaki, I fixed the ( .svp ) file to include the skin with the help of your character model for debugging. I thought about adding skin to ( .svp ) after mix and match skin implementation.
I could have updated sooner but the hot and humid weather delayed me.
Later on the day, after work, I did some experimentation with injecting recorded video into the camera feed. I thought it would be interesting to share. Also, Misaki, I used your model for my test.
My DeviantArt test video because stock video license didn't allow Youtube: https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/1e351697-997f-420a-b339-55dc604511c0/dfayw2p-8bac9980-e12a-4d89-b384-0978995dbef1.1080p.0ddcac221fef40d48b9618a24a4f0d73.mp4
Stock video from: https://mixkit.co/free-stock-video/closeup-of-young-woman-thinking-about-decision-15776/
Misaki, I fixed the ( .svp ) file to include the skin with the help of your character model for debugging. I thought about adding skin to ( .svp ) after mix and match skin implementation.
Great! This is really helpful. Thank you!
I uploaded another video of the current model I updated yesterday with a few more modifications (I made a part of the outline of the face and the side hair disappear when turning face left or right.):
Regarding your new experimentation, it is really interesting! It would be possible to record facial footage outdoors and then replace the face with their Vtuber model based on that video at home. Or simply, it would be easy and helpful to use recorded video for testing. (I always had to call my husband to help me when testing multiple face tracking, so it would be useful if I could record our faces and test multiple face tracking whenever I wanted.)
The heat is really severe, so please be careful of heat stroke!
The heat is really severe, so please be careful of heat stroke!
Yes I am staying cool. Thank you for your concern.
Great! This is really helpful. Thank you!
You are welcome!
Regarding your new experimentation, it is really interesting! It would be possible to record facial footage outdoors and then replace the face with their Vtuber model based on that video at home. Or simply, it would be easy and helpful to use recorded video for testing. (I always had to call my husband to help me when testing multiple face tracking, so it would be useful if I could record our faces and test multiple face tracking whenever I wanted.)
Yes, I plan to include uploading video file for Prototype. I still need more time to work on this feature.
Erika on Discord
[/i][/b][/u]
If only the yes jitter could be fixed, this could be usable! ? maybe with an average slider that the user can self adjust?
Animation Smoothing Function Test. I removed the low-pass jitter filter and replaced it with an unweighted moving average smoothing.
FWIW, I like this smoothing function (based on ola.js):
function Smooth(f,t){var p=performance,b=p.now(),o=f,s=0,m,n,x,d,k=f;return function(v){n=p.now()-b;m=t[i]t/n/n;d=o-f;x=s[/i]t+d+d;if(n>0)k=n<t?o-x/m/t[i]n+(x+x-d)/m-s[/i]n-d:o;if(v!=void 0&&v!=o){f=k;b+=n;o=v;s=n>0&&n<t?s+3[i]x/m/t-(4[/i]x-d-d)/m/n:0}return k}}
It is very easy to use:
// Setup: initial value, milliseconds over which to smooth
var smooth = new Smooth(0, 350);
// Then later, use the smoothed value returned by this instead of the raw value:
var smoothedValue = smooth(rawValue);
It gives a smoothed value that moves to the raw value over time. It's better than the usual interpolation between a start and end value because it dynamically adjusts the curve smoothly when the raw value is changing a lot. You can see it in action here. Try changing the time from 1000 to 1500 (click Run afterward) for a more interesting demo, showing what happens when the raw value hasn't been reached yet and the raw value changes.
The updates since I last tried are awesome! Very excited about the timeline animation addition for getting inbetweens looking nice. Switching attachments, mesh deformation on eyelids, color changes, all that seems to work great. The skins selector is also nice to have. It does seem smoother too after the smoothing code was added, despite the fact that my lighting is very dim right now and my camera isn't picking up my face too well. :upsidedown: I'm going to make a more complex character rig and see how refined I can get with this. Hopefully I can get that done sometime this weekend. Great work!
I have updated my model, and uploaded a new video using it:
(I noticed after I finished recording that an attachment for the mouth had scaled weirdly and was visible in the front of the back of the hair… This is just my fault.)
The new model just replaced the hair and modified the shape of the face to make it look better, but while I was at it, I did a test of multiple face tracking again. At 0:58 in the video, the index of the face has been replaced, but otherwise it was fine and fun
1.0.7
https://silverstraw.itch.io/spine-vtuber-prototype
https://itch.io/t/2255285/change-log
FWIW, I like this smoothing function (based on ola.js):
It is very easy to use:
It gives a smoothed value that moves to the raw value over time. It's better than the usual interpolation between a start and end value because it dynamically adjusts the curve smoothly when the raw value is changing a lot. You can see it in action here. Try changing the time from 1000 to 1500 (click Run afterward) for a more interesting demo, showing what happens when the raw value hasn't been reached yet and the raw value changes.
Thank you for the information, Nate.
The updates since I last tried are awesome! Very excited about the timeline animation addition for getting inbetweens looking nice. Switching attachments, mesh deformation on eyelids, color changes, all that seems to work great. The skins selector is also nice to have. It does seem smoother too after the smoothing code was added, despite the fact that my lighting is very dim right now and my camera isn't picking up my face too well. :upsidedown: I'm going to make a more complex character rig and see how refined I can get with this. Hopefully I can get that done sometime this weekend. Great work!
Thank you, Cydoni. Keep me posted with your progress on your character rig. I hope you get some time and not distracted 
. Also, congrats on being able to have your wedding ceremony.
(I noticed after I finished recording that an attachment for the mouth had scaled weirdly and was visible in the front of the back of the hair… This is just my fault.)
The new model just replaced the hair and modified the shape of the face to make it look better, but while I was at it, I did a test of multiple face tracking again. At 0:58 in the video, the index of the face has been replaced, but otherwise it was fine and fun
The model came out awesome. The hair is detailed with depth when the head turns. I noticed the mouth part sticking out at the back of the neck . I don't know if I could fix the face index when the face tracking looses focus on one of the faces. There might be a work around but I haven't figured it out yet. I am still working on trying to integrate uploading a video for face tracking. The prototype 1.0.7 has face tracking smoothing so the models should have a smoother animation. The jitter filter was not working that well.
Can a monkey vtube?
Wow, ha ha, your tool can even track the monkey's facial movements, awesome! :grinteeth:
I am glad you are using my new model so soon!
By the way, I was going to try 1.0.7, but I got an error and failed to load the resource:
Failed to load resource: the server responded with a status of 403 ()
Can you think of what might be causing it?
Misaki wroteWow, ha ha, your tool can even track the monkey's facial movements, awesome!
I tried a cat before but it didn't work too well. Primate facial characteristics are similar to a human so I thought the face tracking AI model might be able to classify a monkey's face.
Misaki wroteI am glad you are using my new model so soon!
I was researching constraints for future physics constraint. My test model didn't use any constraints but your model does.
Misaki wroteBy the way, I was going to try 1.0.7, but I got an error and failed to load the resource:
That is from my experimentation with loading a recorded video and face tracking for future update. I didn't include the mp4 file in the folder. Prototype should work without breaking.
I tried the 1.0.7 this morning and the skeleton loaded fine. I never tried to upload a video yesterday, just a skeleton file as usual, but perhaps the problem I encountered yesterday might be a server-side problem. Anyway, it certainly seems to be able to move smoother!
I have also made some minor modifications and uploaded the latest model here:
chara-for-Spine-Vtuber-Prototype.zip
The main modification this time was to separate the eye parts into smaller pieces and adjust the weights.
Misaki wroteI never tried to upload a video yesterday, just a skeleton file as usual, but perhaps the problem I encountered yesterday might be a server-side problem.
It was something I forgot remove when I updated the folder on itch.io.
Misaki wroteAnyway, it certainly seems to be able to move smoother!
It sure does.
Misaki wroteI have also made some minor modifications and uploaded the latest model here:
Downloaded.
In the next update, the *.svp file will include saving information about the size of the canvas, the background color of the canvas, position and zoom of the viewport. I want to include the ability to use a video for face tracking since OatmealMu on Discord asked about the feature.
https://silverstraw.itch.io/spine-vtuber-prototype
Update 1.0.8
* Allow spaces in Spine animation names to be substituted with hyphens ( - ) or underscores ( _ ).
* Reduce draw face mesh execution time from 5-7 milliseconds to 0-1 milliseconds. Increase frame rate for Spine model while face mesh is shown.
* Expand .svp JSON file to include exporting/importing viewport position, viewport zoom, Spine model canvas height and width, Spine model canvas background color.
@[削除済み]
Great, it's very convenient that the viewport and canvas settings are now saved in the .svp file! Also, I am very happy that the improved frame rate allows for quite smooth movement
However, when I look at the console in the developer tools of Chrome, there seems to be some kind of error immediately after opening the page. I am not sure if this has anything to do with it, but sometimes when I try to display the second model, the models turn black. Here is a video of the errors:
I have attached a screenshot of the error log as it may be difficult to see in the video:
This is not a problem when a single model is to be rendered. Could you check the errors?
The HTTP Error 503 is itch.io website issue and should not be caused by Prototype files. lib.js is itch.io file that is added on all web browser games on the website.
I was not able tor reproduce the black shader bug using two models on Chrome. Can you check if it is reoccurring bug? Also, try to load the model files without a .svp file. I wonder if the expansion created some bugs for multiple model loading, but if you did not have an issue with single model then you should not have an issue with multiple models.
The "sample.mp4" is just my testing video that was not included in the Prototype folder. I should be able to include video face tracking in the next update.
Thank you for your reply! I understand each of the error logs.
Can you check if it is reoccurring bug? Also, try to load the model files without a .svp file. I wonder if the expansion created some bugs for multiple model loading, but if you did not have an issue with single model then you should not have an issue with multiple models.
I tried again and confirmed that I can reproduce the bug without the .svp file. Here is the screen recording:
Do you have any idea what might be the cause?