Visym Collector - Frequently Asked Questions (FAQ)

Frequently Asked Questions (FAQ)

What is Visym Collector?

Visym Collector is a mobile app designed for consented and privacy preserving dataset collection for AI applications.
We pay collectors to use our mobile app to submit "collections", which are videos with labeled activities and objects used for training visual AI.

How do I become a paid collector?

We are regularly running collection campaigns. Send us an email at info@visym.com expressing your interest in joining the collection team and we will reach out with next steps when a new collection is available in your area.

Can I use Visym Collector if I am not a paid collector?

Yes! We provide comprehensive python tools to allow you to define your own collections on your device so that you can collect your own data.
We also provide Dropbox integration so that you can link your Dropbox and the videos and annotations you collect will be available for your use.
These tools allow anyone to create their own collection campaign to record, annotate and download videos for ground truth visual AI. If you want to scale up your collection to leverage our worldwide collection team, please reach out at info@visym.com and we will work with you.

How do I get started as a collector?

Read the quickstart document for new collectors for instructions for downloading and using the mobile app.
Try out the Practice collections for gaining experience with the UI.
Join the community for discussion with other collectors and annoucements by the collector team.
Look out for periodic emails from the collector team (collector@visym.com) with your dashboard containing ratings and payments for your videos.
Keep up the good work!

I can't log-in!

Check that there are no extra spaces at the beginning or end of your email address
Check that there are no typos (e.g. gmail.con) and that you are using the email address you used to register without any additional capitalization.

How do I use the editor to correct a mistake during collection?

We rely on the collector community to submit us videos with accurate annotations in the form of bounding boxes for objects and start and end times for activities. If you make a mistake in annotation, you can fix it after you record using the video editor.
Read the video editor tutorial document for new collectors for instructions on how to use the video editor to correct object boxes and activity times.
Watch the tutorial video from the collector team. Please enable closed captions to improve your experience.
We recommend that you submit your video, then edit it from your Video feed in-app, rather than editing after collection and prior to submission.

What happens to the videos after I submit them?

Collector is a platform to collect and curate video datasets for visual AI. When you submit a video to us, we prepare it for use by the research community for training a visual AI system, in the form of videos available for public download.
An example of a publicly released dataset containing videos provided by the worldwide collection team is People In Public.

How do I keep track of the videos I have submitted?

When a collector is onboarded you will be sent a welcome email that contains a link to your collector dashboard. This dashboard contains all of the videos that you have submitted to us, along with the ratings from the review team. You can use this to track your expected payments for the week.
Your dashboard is also available by logging using the same email and password that you use in-app at https://collector.visym.com
Please note that your dashboard updates approximately every few hours, so there will be a lag between when you collect and when you see this updated. Submitted videos are available for review immediately in-app. Reviews for your videos are usually available within 24 hours.

Why did the collections available in the app change?

We are regularly updating the collections and projects available in the mobile application. We may enable or disable collections on demand, based on the needs of our collection, and you may see this as collections dissappearing from your app, and replaced with new collections. This is expected.
We usually update the collections around 8am ET daily.

What do the "Activity Ratings" mean in my collector dashboard?

Each video that you submit is looked at by our review team and assigned one or more ratings according to our review criteria. The activity ratings are a short summary of the output of this review procedure, where the descriptions of the ratings are provided below.
Good. This activity was rated by the review team and passed our review criteria.
Perfect. This submission passed our review criteria, and is an excellent example of this collection which can be used to train other collectors. Great job!
Rate limit exceeded. Each collection can be submitted as many times as you choose, however we will use for payment only the last video submitted for each collection type within a specified time window. We do this to enforce diversity so that we get videos that are separated across days. For example, if the rate limit is 12 hours, you may submit a video for collection "Ride a Bicycle" at 6pm, then again at 6am the next day.
Incorrect activity count (M<N). Each collection has a specified number of activity buttons. We require that you press each activity button at least once. Please review your videos carefully before submission, and retake them if there are an incorrect number of activities present. The review (M<N) refers to the fact that your video contained only M activities, but we expected at least N.
Incorrect timing. The start time or end time of the labeled activity is not precise enough. Please review your videos so that the activity starts just after the button is pressed, and the activity ends just before the button is pressed again. We require that at least 50% of the activity is present within this temporal window. If we cannot tell that the activity occurred within the quicklook summary, then it will be flagged as incorrect timing.
Box not centered/Box too big/Box too small. The bounding box surrounding the actor performing the activity is either too big, too small or not properly centered on the actor. We require that the actor is completely inside and covers at least 50% of the box. We would prefer if you err on a box slightly too big with the actor completely inside rather than a box too small with the actor patially inside and partially outside the box. The ideal box is one that is tight around the actor both in width and height. You can change the box by dragging the corners in the collection UI.
Incorrect label. The activity performed in the video does not satisfy the collection definition. Please read the definition carefully and try again. This most often occurs with swapping "entering" vs. "exiting", or purchasing from a vending machine in your liviing room.
Object/activity not visible. The activity or the object being used in the activity are not visible in the video. Please make sure that the video is well lit enough that the activity can be seen, and that the activity being performed is clearly visible in the video. If the review team cannot see it in the quicklook, then it will be flagged as not visible.
Repeated scene. Our review team has determined that you are using the same background again and again. This is a warning. We need new viewpoints and new scenes for you to perform your activities.
Incorrect viewpoint. Our review team has determined that you are not collecting from the required camera viewpoint. The viewpoint is specified in the collection description.
Wrong location. The video was collected in the wrong location, such as indoors vs. outdoors. The locations is specified in the collection description.
Invalid repetition count. This video was submitted without completing the required repetitions as defined in the description. A repetition is a repeat of the activity performed slightly differently each time, such that when each activity is performed the corresponding activity button is pressed to specify the activity start, the activity is performed with the button enabled, then when the activity is complete the button is disabled. The subject returns to a neutral body position with the activity button disabled, pauses, then repeats the activity. This review response typically means that either the collector performed only one repetition, or that the multiple repetitions were recorded but each repetition was not performed with separate activity button presses with pauses in between them.
Incorrect instances per video. The video did not contain the required number of activity instances in a single video. This often occurs when you are required to include a certain number of activities in a video, and this minimum number was not met.
Wrong actor pose. The video was collected with the actor not conforming to the required pose, such as sitting or standing. The required pose is specified in the description.
Incorrect scene illumination. The video was collected with the wrong lighting conditions, such as shadowed vs. direct sunlight. The required scene illumination is specified in the description.
Incorrect style. The video was collected with the actor not performing the the required style, such as an adverb like angry or excited. The required style is specified in the description.
Incorrect objects. The video was collected with the actor not interacting with the required object or objects. For example, the required object for Opening a microwave is a microwave. Required objects are specified in the description.
No joint activity. The video was collected without the actor performing a second joint activity performed simultaneously with the primary activity. An example is walking while talking on a cell phone, where the actor is performing the walking activity at the same time as talking on scell phone. The required joint activity is specified in the description.
Unusable video. The video submitted is not usable by the collector team. This is usually due to the video being too dark, your finger accidentally occluding the camera lens or contains a scene with no activities (e.g. the sky or floor).
Needs improvement. The video has something that requires secondary review, such as containing children under the age of 18. We require that all people contributing to the video are 18 years of age or older.
Consented subject not in video. Our review team has determined that the subject reported during consenting is not present in the video. When you are consenting your subject, you must enter the email address and record the consent video for the subject that will be in the video. The review team compares the consent video saved for this subject to the recorded person in the video, and if they are not the same, then the video will not be accepted. A common error is entering your email address during consent rather than the email address of your subject who will be recorded.
Unrated. Our review team has not gotten to this submission yet, but it is in the queue for rating.
Under review. Our review team has not gotten to this submission yet, but it is in the queue for rating.
Incorrect GPS location. The GPS location reported with your submission does not match the required GPS location for this collection. Check that you collected at the right location.
Repeated GPS location. This location has already been submitted. Try to move to a new location at least 200 meters from this location.
GPS accuracy too low. Your device is not receiving accurate GPS positioning from the available sattelites. Avoid collecting near trees, bridges, tunnels or too close to buildings, or try again another time.
Video too short. The required duration in seconds as shown in the collection description is not met.
Outdated App. You are required to upgrade to the latest version available on the app store. This is required so that the additional metadata such as position, orientation and calibration data that that we collect along with the video are accurately recorded. If your device does not support the minimum required version, then you will not be able to collect until you upgrade your device, or borrow a device from a friend (remember to log-in using your collector email!).

How is "Accepted?" computed in my collector dashboard?

Each video you submit is split into the individual activities in the video, and each activity is rated by the Visym Collector review team. The activity ratings are combined into a single score for the whole video, taking into account the accuracy of the bounding box, the accuracy of the timing of the start and end of the activity, and whether the collector followed the instructions in the collection description. This score captures how useful the video is for the collector team. If this number of greater than a collector team specified threshold, then it is accepted for payment.
You can review the collection score in the cell formula contents of your dashboard, along with the target score threshold.

Why do some activities have a rating of "Good" but the "Accepted?" is "NO" in my collector dashboard?

We pay out for complete videos that satisfy our criteria, and not activities. Each video is split into a set of activities, each is individually rated. We combine the ratings from all of our reviewers and from all activities in a video into a single score for the video. This score capture how useful this video is for our machine learning. If this video score is greater than a fixed threshold for the campaign, then the video is paid out.
This means that there may be activities that are good, but videos that are not good as a whole. You can see which activities come from the same video if they share the same Video ID.

How do I get paid?

We calculate payments based on the quality and quantity of the videos you submit to us, and the payments for the week are calculated on your collector dashboard.
Log into your dashboard using the same email and password you use for the app, and update your payment details under Profile.
You can review your expected payments on your dashboard.
We support paying collectors via Payoneer and PayPal. Contact us at collector@visym.com if you would prefer alternative payment arrangements.

How do I use the dashboard at collector.visym.com?

Use the same email and password you use to log into the app. This will show your dashboard which summarizes your submissions and expected payments.
You may use "open in a new tab" button in the top right to access your dashboard as a Google spreadsheet.
Enter your payment details under Profile. We currently support Payoneer and PayPal.