On 2009-08-17 09:18:00, Allan Engelhardt wrote in CYBAEA Journal:
We knew the potential existed already, of course. Mobile devices in the USA generates some 600 billion transactions per day, each tagged with the location and time. Jeff Jonas: Every call, text message, email and data transfer handled by your mobile device creates a transaction with your space-time coordinate[...]. Got a Blackberry? Every few minutes, it sends a heartbeat, creating a transaction whether you are using the phone or not.
That is some 7 million transactions per second, on average.
The mobile operators have this data, of course. We all know this (especially here where we have been using some of it for social network analysis), at least in principle. No real surprises here, except perhaps in the volumes.
But did you know that the operators are sharing your data? What is new, at least to me, is that this data is being provided to third parties that are leveraging specially designed analytics to make sense of our space-time-travel data. With the data out and specialized analytics emerging, this infant industry is already doing some pretty amazing work. Your space-time-travel data makes where you live and where you work self-evident, and it reveals your most frequent, periodic, infrequent and rare destinations.
This is powerful stuff. I can't validate Jeff's claim that the network operators are sharing their detailed network data in any systematic way and for anything other than research purposes, but surely it is only a matter of time. This data is too good to ignore — super-food for analytics, as Jeff calls it.
Think about the possibilities. We obviously know where you live and where you work, but also who is in the house with you (your wife, hopefully) and which of your office mates join you for the Friday afternoon beer.
We know which department stores and malls you visit. We know where in the stores you spend the time. We know your hobbies, your interests, your friends, .... We know your mistress and your bank manager, your colleagues and the boys on the football team you coach. We know where you shop and when you shop, we know your commuting route and that you always sop at Starbucks for a coffee on the way in. We know it in real-time so we know you are going to be late in the office this morning. Would you like us to let them know that you will be there at 09:23 (or 09:12 if you do not stop for that coffee on the way)?
There are so many possibilities. Much good could be done with this data. Some bad. Which service would you like to see developed?
On 2010-03-08 14:46:00, Allan Engelhardt wrote in CYBAEA Data and Analysis:
I needed a fast way of eliminating observed values with zero variance from large data sets using the R statistical computing and analysis platform. In other words, I want to find the columns in a data frame that has zero variance. And as fast as possible, because my data sets are large, many, and changing fast. The final result surprised me a little.
Read more (~501 words).
On 2009-08-17 09:18:00, Allan Engelhardt wrote in CYBAEA Journal:
We knew the potential existed already, of course. Mobile devices in the USA generates some 600 billion transactions per day, each tagged with the location and time. Jeff Jonas: Every call, text message, email and data transfer handled by your mobile device creates a transaction with your space-time coordinate[...].
The mobile operators have this data, of course. We all know this (especially here where we have been using some of it for social network analysis). No real surprises here, except perhaps in the volumes.
But did you know that the operators are sharing your data? What is new, at least to me, is that this data is being provided to third parties that are leveraging specially designed analytics to make sense of our space-time-travel data.
Read more (~449 words, 1 comments).
On 2009-07-27 19:38:00, Allan Engelhardt wrote in CYBAEA Data and Analysis:
O'Reilly's recent publication Beautiful Data has a chapter by Jeff Jonas which is enough reason in itself for me to recommend it. The chapter, Data Finds Data, is also available as a PDF download.
Read more (~66 words).
On 2009-07-22 13:37:00, Allan Engelhardt wrote in CYBAEA Data and Analysis:
This is by far the best description of why traditional parallel databases (like Teradata, Greenplum et al.) is a evolutionary dead end. But much more than a theoretical discussion, they have built a solution which they call HadoopDB. It is based on Hadoop, PostgreSQL, and Hive and is completely Open Source. Alternative, column-based, backends to PostgreSQL are being implemented now. Read: Announcing release of HadoopDB.
Read more (~83 words).
On 2009-07-22 06:59:00, Allan Engelhardt wrote in CYBAEA Journal:
The nice people at Velocity has released The B2B Content Marketing Workbook. It is behind a registration wall which means we wouldn’t normally recommend it but you can just type junk in the fields if you are not comfortable with giving your personal details to a marketing agency. (Think about it....) If you are relatively new in the B2B world, say having joined a professional services or consulting organization, you may find this one useful.
Read more (~263 words).
Join the discussion
To good to be true ...
Allan, I don't think I can fully agree to your last paragraph though. As much as MNO's know from the above data, do you think that this kind of analysis will be allowed in Europe? Even now tracking on travelling needs to be anonymized if can be used for marketing purposes at all.
Besides the fact you need a real big beefy machine to analyze all this data, right? What you want to combine is SNA from CDR's with location & time data using triangularization. Wow, it's an appealing idea but like always what is the financial benefit to justify such investment and later on knowledge?
Where does your marketing campaign comes in? Ok, you might know which MSISN'd I buddy in for a beer on a Friday regularly (well, maybe once a month), you know that a couple of MSISDN's are in the same sporting complex that I'm in twice a week, but how are you going to make some money out of this?
Just a little challening you :-)
Andreas