In this paper we discuss the design and evaluation of a mobile based tool to collect activity data on a large scale. The current approach, based on an existing activity recognition module, recognizes class transitions from a set of specific activities (for example walking and running) to the standing still activity. Once this transition is detected the system prompts the user to provide a label for their previous activity. This label, along with the raw sensor data, is then stored locally prior to being uploaded to cloud storage. The system was evaluated by ten users. Three evaluation protocols were used, including a structured, semi-structured and free living protocol. Results indicate that the mobile application could be used to allow the user to provide accurate ground truth labels for their activity data. Similarities of up to 100% where observed when comparing the user prompted labels and those from an observer during structured lab based experiments. Further work will examine data segmentation and personalization issues in order to refine the system.