State Department database crash strands hundreds at the border

Outage caused by corrupted data, which was mirrored to backup—making it useless.


by Sean Gallagher - Jun 23, 2015 11:14am PDT  Ars Technica

Thanks to the failure of a system used to collect and transmit fingerprints and photos, the US State Department has been unable to issue visas to travelers or guest workers for the past two weeks, The New York Times reports. While some of the systems related to visa processing have been restored, biometric information is still not being processed, leaving many travelers from outside the US and hundreds of agricultural "guest workers" stranded.

State Department officials told the Times that the issue was related to a hardware failure. In an e-mail to the paper, Consular Affairs spokesperson Ashley Garrigus said that there had been data corruption caused by a hardware failure, which had been replicated to the biometric database's backup system. “While switching to the backup system," Garrigus wrote, "we discovered that the data was damaged and unusable. We deeply regret the inconvenience to travelers and recognize the hardship to those waiting for visas, and in some cases, their family members or employers in the United States.”

The State Department normally processes about 50,000 visa applications a day, according to astatement on the outage on the agency's website. But that number surges seasonally as employers bring in laborers (mostly from Mexico) to harvest crops. On top of that, the overall number of visas annually has grown. Earlier this year, the State Department put out a "sources sought" call for a new facial recognition service because of the huge growth in visa requests, especially those driven by the H2-A and H2-B visa programs. "At present, the Visa and Passport face galleries contain over 275 million images combined, which are among the largest face recognition datasets in the world and growing annually at an anticipated rate of 23 million images per year," the State Department's procurement officer wrote in the procurement announcement . "Every passport and visa application requires a face recognition search of the legacy and watch-list galleries."

The guest worker visa program itself is a tangled web of systems. First, employers apply through the Department of Labor with candidate workers. Then the US Citizenship and Immigration Service at the Department of Homeland Security screens the individuals petitioning through the employer for visas. USCIS next sends approved petitions to the State Department’s Kentucky Consular Center to be entered into the visa database. And that in turn allows consular and embassy offices outside the US to see that the individuals are authorized to apply for a visa, thus submitting biometric data back to the Kentucky Consular Center's database to check against watch lists and other image and fingerprint data that might catch attempted visa fraud.

Because of the system failure, the State Department has delayed visas for over 1,500 guest agricultural workers in Mexico. "Last week, nearly 1,250 temporary or seasonal workers who had been issued visas in the past were issued new visas in Mexico," a State Department official said in a statement on the outage. "We have issued more than 3,000 visas globally for urgent and humanitarian travel."

According to a Reuters report, the outage is having real economic impact—especially in Washington, where cherry growers have been unable to get workers out to pick before fruit becomes unsellable. The blueberry crop may be affected next, as workers usually stay to harvest them after cherries. Washington Farm Labor Association Director Dan Fazio told Reuters, "Our farmers are all in for the guest worker program, but the government isn’t. We have a lot of cherries that are ruined and it looks like a lot of blueberries are going to be lost.”

15 Massive Online Databases You Should Know About

Make Use Of

Written by Gavin Phillips16 June 2015

Think of your favorite open databases.

I’m sure Wikipedia and IMDb instantly spring to mind, but you might not be in the need of all that knowledge ever, or a comprehensive database of all things entertainment. Sometimes you need a bit of VLDB (Very Large Data Base) flavor. Something to spice up your data analysis. Something to put the “big” in your big data. Whelp, good person, you’re in the right place.

Here are 15 massive online databases you can access and analyze for free, or just peruse at your leisure.

1000 Genomes

The 2003 completion of the Human Genome Project (HGP) was just the beginning. Since then advances in sequencing technology have vastly reduced the per-person cost allowing vast expansion of the HGP from its initial research base of twenty university labs, into a sprawling, globalized network of interconnected genome mapping facilities.

You can download part of the 1000 Genomes Project, containing sequencing information for over 2,600 people from 26 populations around the world. This is a 200TB file, so be prepared. We would suggest using it in conjunction with a powerful cloud computing platform.

See also: The Animal Genome Size Database for genome data relating to 5635 species.

Airliners

The planespotters heaven. A massive image database featuring 2,532,457 photos of all manner of aircraft, from the smallest individual craft to hulking great flying fortresses.

Airliners also features an extensive aircraft data and history section always kept updated in cooperation with Aerospace Publications to ensure factual accuracy. This has made it one of the single most detailed aircraft databases on the Internet.

See also: Try Planespotters.net for a different range of images, or SeatGuru for airplane seating schematics.

The Internet Archive

The site formerly known as The Internet Archive, has gone through a massive redesign. The site hadn’t changed much since around 2002, but a lot has changed since then. The Internet Archive has done even more growing since the early days.

Archiving everything on the Internet, the site gives you free access to digital media including books, music, games, videos, and much more. The collection is currently estimated at around 10 petabytes, and as their webcrawlers keep crawling, it will continue to grow.

Freebase

Freebase is “a community-curated database of well-known people, places, and things,” stored in a data structure called a graph. A graph is comprised of nodes, connected by their edges, which allowed Freebase to rapidly expand its content without disrupting existing records.

Unfortunately, Freebase, owned by Google, switched to read-only mode early this year, before the standalone service database is transferred over to the Wikimedia Foundation for integration in the Wikidata project (end of June, 2015). Developers can currently still access Freebase using existing APIs, but once the switch is made, developers will have to use Wikimedia APIs to access the data.

Find a Grave

From the home base of an Internet knowledge dream-team of Google and Wikimedia, we move to the morbid. Find a Grave is a massive, 121 million record database of burials around the globe.

Most comprehensive records come from the US, but there are some smaller countries with large data. Complete with photos, interesting monuments, and a number of interesting epitaphs…if you need inspiration?

GameRankings

A database maintained by the ever-present reviewing team at Gamespot. GameRankings gives a well-rounded portrayal of a game’s popularity by covering on-and-offline gaming reviews from reputable sources.

The Big Cartoon Database

In a similar vein to the massive IMDb, The Big Cartoon Database focuses exclusively on all things animated: cartoons, films, television shows, adverts, and more. If it is an animation, you’ll find it here – and if not, sign up as contributor to this ever growing database.

The Big Cartoon Database has a sister site in The Big Comic Database, home to a further 100,000 or more comic book records, spanning some 5,000 series, with over 35,000 cover scans. It also contains a comprehensive search function, including a comic book price guide detailing current resale values at the various grading levels.

See also: The Grand Comics Database, a non-commercial enterprise database of comics worldwide.

CiteSeerX

An invaluable tool for students and academics alike, CiteSeerX is a public search engine and digital library of academic and scientific papers. Often considered the first automated citation indexing system, it was the inspiration for Google Scholarand Microsoft Academic Search. Though the latter has since been integrated into the Bing search engine.

CiteSeerX focuses on indexing public scholarly documents. If your research paper is openly distributed, it has a higher chance of appearing within the search engine. CiteSeerX is an excellent example of the power of shared knowledge made available to a much wider audience.

See also: Google Scholar for a different range of books and citations.

WorldCat

Unfortunately not a database of each cat picture on the Internet. Now that would be something! WorldCat is much more useful than that.  The reference site documents the collections of over 72,000 libraries around the world, covering 170 countries and territories. This is useful if you’re researching in a foreign country, or just have a desire to read rare books in person.

The only downside is the update method. WorldCat uses a batch processing model rather than allow users to access the data in real-time. So, WorldCat does not indicate the loan status of catalogued books, whether a library owns multiple copies of one book, or whether the book in question is directly accessible to those wishing to visit. It is still a very useful tool, especially when used in conjunction with CiteSeerX.

The Simpsons Archive

“The Internet’s clearinghouse of Simpsons guides, news, and information.” I couldn’t have put it better myself. The long-standing fan favorite began way back 1994, and is still going strong even without any interactive multimedia, if only to escape the watchful eye of Fox’s legal department.

WinCustomize

You will find one of the single largest databases of customization tools for Windows here, spanning from XP up to Windows 8.1. I’m sure it won’t take long for Windows 10 to begin making the rounds. Its vast popularity stems from a combination of forces. Owner Stardock, subsidizes the site meaning there are little-to-no advertisements. It also benefits from the number of individuals funneled to the site from Stardock.

Ultimate Guitar Archive

Ah, a trip down nostalgia lane to a database reminding me I was never to be Roger Waters. In fact, I can still barely play, but that’s another story.

The Ultimate Guitar Archive, or just Ultimate-Guitar (UG), has over 1,500,000 registered members around the world, overseeing a ridiculously large amount of community content. It is almost mind-boggling how much guitar related information is scattered out from a single source. The community just doesn’t maintain a massive database, they also frequently collaborate with one another to create sprawling music projects.

Plants for a Future

Plants for a Future documents  ecologically sustainable horticulture. It has a big hand in spreading knowledge on species diversity and the importance of permaculture. What started as a small project in the depths of Cornwall has slowly grown into a worldwide database.

Growth is somewhat slow, and largely focuses on permaculture in the UK and EU, but many of the records can be swapped for specific locations in the US once you have the species details.

Quandl

Power up with this Excel add-in to process and analyze data. The main Quandl site acts as a database search, locating databases from around the world that match your search terms. Try it if you need some extra data in a hurry, or just like playing with large datasets (honestly, who doesn’t?!).

See also: The Enigma database search engine.

Tiny Images

The Tiny Images dataset acts as a visual dictionary. Click anywhere within the image and a search term pops up with extra information. You can also use specific terms to sift through 80 million images.

The database is part of wider machine learning project focused upon teaching computers to “see” and “read” semantic fields within images.

Bonus Source: /r/datasets

The “front page of the Internet” is a solid home for data mining enthusiasts around the globe. There are subreddits dedicated to machine learning, data mining, text to data, and datasets. If you need something specific make a request. New datasets appear every week.

Watch out for the interesting datasets posted like the Immunization Levels in Child Care and Schools for California.

Do You Use The Wealth?

The Internet has created the single-clearest opportunity for individuals to come together and concentrate their knowledge into a single database. We are valiantly trying to document everything about anything. Some of these databases are for perusing, others are for learning, but we hope you enjoy them all.

What are your favorite databases? Are there any open massive reference sources I should have included in this list?

Image Credits: network server via Shutterstock, library via Shutterstock

The Data Economy is a Rich Source of High-Paying Jobs

April 3, 2015 by Joshua New Masters in Data Science


Last month, the U.S. Department of Commerce published a report that found data is fueling a rapid growth in high-paying, high-skilled jobs in the public and private sector and transforming the United States’ occupational landscape. The report, titled “The Importance of Data Occupations in the U.S. Economy,” defined “data jobs” as occupations which place high importance on analyzing data, processing data (i.e. coding and auditing), and interacting with computers (i.e. writing software and performing data entry) and found that there were 10.3 million of these jobs in 2013 (about 7.8% of all employment in the US). The report acknowledges that this is a conservative estimate — if the criteria were broadened to include jobs that describe these activities as just “important,” instead of “very important,” the category would include 64 million jobs, or 59% of total employment in the United States.

What’s Driving This Job Growth?

The increased use of data in the private sector has been one of the leading drivers of job growth in the past decade. Of the 10.3 million data jobs identified in the report, 8.6 million are in the private sector. These jobs are highly paid, with an average hourly wage of $40.30 in 2013 — 68% higher than the average wage for all of industry. From 2003 to 2013, the private sector industries that are considered data-intensive (meaning that at least 25% of their employees are in data jobs) increased their number of data jobs by 21%, compared to the overall average job growth of 5% for this period. Additionally, these data-intensive industries grew faster during the economic recovery period of 2010-2013, seeing a 9.4% increase in employment, compared to the 6.2% overall average. Any attempts at enacting broad-based privacy legislation that would limit data collection could hinder the potential for the private sector to find new applications for data, and thereby create jobs to act on these opportunities.

What Kinds of Workers and Skills Are Required?

Data workers were found to be considerably more educated than average. Nearly 70% have a bachelor’s degree or higher. By comparison, just about 35% of the overall workforce has that level of education. To sustain the growth of the data economy, the United States needs a workforce that is educated to meet the needs of companies that want to innovate and grow in the United States. The labor market is already struggling to meet this demand, creating a data science skills gap, and this problem will only grow worse if educational institutions on all levels do not adequately equip the emerging workforce with the necessary data skills.

What Should We Do About It?

To ensure that the economy’s growth is not hindered by a lack of qualified workers, federal and state lawmakers should promote data science education at all levels. Policymakers should fund efforts to expand enrollment in statistics and computer science courses and support the development of massive open online courses that teach data skills, and are accessible to both those in school, and those already in the workforce seeking to become data-literate. Secondary schools can also help develop a pipeline of future data-literate workers by creating more flexible math requirements, enabling students to satisfy them by taking computer science or statistics courses instead.

It is important to note that this report uses data from 2013. The trend towards more data-driven industry and the creation of more data jobs has no doubt continued, if not accelerated, since then, driven in part by new technologies like the Internet of Things. As companies increasingly rely on data to improve operations and develop new products, there will be an even greater need for policymakers to develop a workforce that can make use of all this data and, and for them to craft policies that allow such technologies to flourish.

Joshua New is a policy analyst at the Center for Data Innovation. His research focuses on methods of promoting innovative and emerging technologies as a means of improving the economy and quality of life. The Center for Data Innovation is a non-profit, non-partisan think tank studying the intersection of data, technology, and public policy. Based in Washington, DC, the Center formulates and promotes pragmatic public policies designed to maximize the benefits of data-driven innovation in the public and private sectors.

Cisco Western Academy Conference: Call for Presentation Proposals

WASTC Cisco West Coast

Academy Conference 2015

June 18 & 19

Call for Presentation Proposals!

Hello Greatly Talented People!

It is time to submit your presentation proposals for the Academy Conference 2015!  Collaboration is the name of the game, and we all need to hear about what you do well.  Don’t be modest—you have lots of tools in your bag and we would love to [some of us need to] hear about one or two of them.

You will have 75 minutes for a presentation on either June 18^th or 19^th at Cal Poly Pomona.  

• Technical
• Educational Pathways
• Jobs
• Industry
• Student Competitions
• To infinity and beyond…

Also if you would like to hear a presentation on a specific topic, please send that request and we will try to make it happen.

To submit your proposals please send me:

1.    Your Name

2.   Title

3.   Short presentation description

  

Come show us what you got!

  

General Conference Information

Pre-Conference

June 15 – 17

Academy Conference

June 18 - 19

·      36 short sessions—full descriptions will be posted at a later time on the WASTC site
·      2 Key Notes
·      Appreciation Event (live band, food, and industry contacts)

Cost for one or both conferences

            $99 for WASTC Academies

            $150 for all others

To register and find more information, see the WASTC website: www.wastc.org

Best regards, Karen

--

Karen Stanton
Cisco Networking Academy
ASC ITC Ohlone
WASTC Coordinator
Western Academy Support & Training Center-WASTC
www.wastc.org
661 362 5363
karen.stanton@wastc.org

MPICT

www.mpict.org

Big Data, Hadoop Standards Group: Who's In, Who's Missing?

by JOE PANETTIERI

FEB 19, 2015 1:12pm ET

All eyes in the big data world are on the Open Data Platform -- a new association that strives to promote big data technologies and open source platforms like Hadoop. While promising and backed by big names like GE and IBM, the Open Data Platform initiative also lacks some key names. Here's a reality check.

First, the big picture. The Open Data Platform will "promote big data technologies based on open source software from the Apache Hadoop ecosystem and optimize testing among and across the ecosystem’s vendors. These efforts will accelerate the ability of enterprises to build or implement data-driven applications," according to a statement from the association's founders.

In decades past, similar standards groups have united to promote Linux, Unix, WiFi and other emerging platform technologies. But standards groups can also resemble the political landscape -- as vendors sometimes break off and move to the extreme left or right of the group's stated goals.

Big Names to Start

Several industry giants and startups are driving the Open Data Platform group -- including Altiscale, Capgemini, CenturyLink, EMC, GE, Hortonworks, IBM, Infosys, Pivotal, SAS, Splunk, Teradata Verizon and VMware.

Still, some key names also are missing from effort. Chief among them:

• Cloud services providers like Amazon Web Services, Google Cloud Platform, Microsoft Azure and Rackspace -- each of which promotes various Hadoop efforts on the public cloud.
• Hadoop specialists like Cloudera (which now has a $100 million annual revenue run rate) and rival MapR -- both of which compete with Hortonworks.
• Hardware providers that ship servers and tune their systems for Hadoop -- including HP, Dell and others.
• NoSQL database providers that work closely with the Hadoop industry -- such MongoDB and others.

Both Cloudera and MapR have publicly stated that they consider the Open Data Platform effort to be redundant with existing Hadoop standards work. And some critics view ODP as an EMC-VMware-Pivotal effort to boost the trio's relevance in the Hadoop game.

Eight Core Goals

Despite those points of debate, the Open Data Platform's powerful members should be able to flex some muscle in the weeks and months ahead. The existing members say they have eight core goals:

1. Accelerate the delivery of big data solutions by providing a well-defined core platform to target.
2. Define, integrate, test and certify a standard "ODP Core" of compatible versions of select big data open source projects. This area, Information Management believes, could be particularly tricky as vendors potentially try to promote their wares into the standards-based platform.
3. Provide a stable base against which big data solutions providers can qualify solutions.
4. Produce a set of tools and methods that enable members to create and test differentiated offerings based on the ODP core.
5. Reinforce the role of the Apache Software Foundation (ASF) in the development and governance of upstream projects.  This is particularly important, Information Management believes, since Apache has been so instrumental in Hadoop's development so far.
6. Contribute to ASF projects in accordance with ASF processes and Intellectual Property guidelines. Here again, Information Management believes the group is trying to stress open, vendor-neutral collaboration through ASF.
7. Support community development and outreach activities that accelerate the rollout of modern data architectures that leverage Apache Hadoop.
8. Will help minimize the fragmentation and duplication of effort within the industry.

That last point could be particularly difficult for ODP members to navigate, Information Management believes. By their very nature, technology companies strive to differentiate their wares through proprietary, high-value add-ons.

But on the flip side, there are numerous examples of open standards -- HTTP, Ethernet, WiFi, etc. -- that ultimately benefitted both vendors and their customers. Open Data Platform certainly hopes its efforts mirror those hugely successful outcomes.

RHT: Data Analyst: Making Data Actionable

by Robert Half Technology
July 21, 2014


The average starting salary for a data analyst is expected to increase 5.3 percent in 2014.

The data analyst is responsible for helping to distill an organization’s technical data in a way that is meaningful to business decision makers.
Christina Giglio, branch manager for Robert Half Technology, says raw data would be “useless” to nontechnical users without help from data analysts. “Technical information needs an interpreter,” she says.

Data analyst salaries due to increase in 2014
Skilled data analysts are in demand, according to Giglio, and many leading employers are increasing starting compensation for this role. Robert Half Technology’s 2014 Salary Guide reports that the average starting salary for a data analyst in the United States is expected to increase 5.3 percent in 2014, ranging from $67,750 to $101,000.* (You can use the Robert Half Technology Salary Calculator to find salary information specific to your city.)

Becoming a data analyst
If you’re thinking about exploring a data analyst career, here are some of the qualifications that potential employers will be looking for:

• Several years of experience working with major database platforms, such as Microsoft SQL Server, Oracle and IBM DB2
• Strong analytical, quantitative and problem-solving abilities
• A thorough knowledge of relational database theory and practice
• A bachelor’s degree in computer science, information systems or a related field
• The ability to work both independently and collaboratively with data systems teams
• Strong verbal and written communication, analytical and problem-solving skills

Giglio says that an ideal candidate for the data analyst role will have at least five years of experience with accounting or financials. For students who would like to pursue a data analyst career, she suggests obtaining degree in computer science or statistics and taking courses in math, data integrity and data modeling. For aspiring data analysts already on the hunt for entry-level positions, Giglio says, “Look for opportunities working with databases such as Microsoft SQL Server to get the key skills needed for your resume.”

The responsibilities of the data analyst
Data analysts are responsible for analyzing complex data systems and documenting data elements, data flow, relationships and dependencies. In addition, they are often tasked with developing automated and reusable routines for extracting requested information from database systems. Data analysts also will usually work in partnership with business analysts, data architects anddatabase developers to build data transaction and warehousing systems.

Compiling detailed reports using data reporting tools such as Crystal Reports is another typical responsibility of the data analyst. “For this role, candidates need to be skilled in the tools used to retrieve and manipulate data,” says Giglio. “An understanding of the Microsoft stack of technologies and the Microsoft .NET development system can be helpful.” She adds that experience with Microsoft SQL Server, Oracle or IBM DB2 is also important for data analysts because these types of tools are essential for actually working with the data.

While the data analyst job description is similar to that of the business intelligence analyst, data analysts do not typically play a part in making business decisions — although management may ask them to make recommendations based on their findings, says Giglio.

Look to Robert Half Technology’s latest Salary Guide for job descriptions and starting compensation trends for a wide range of IT jobs — including data analyst. Want to learn about other types of analyst roles in IT? See this post about data warehouse analysts.

*Data analyst/report writer salaries in Canada are expected to increase 5.1 percent in 2014, with starting compensation ranging from $72,250 – $99,250. Figures are in Canadian dollars.



— Robert Half Technology

With more than 100 locations worldwide, Robert Half Technology is a leading provider of technology professionals for initiatives ranging from web development and multiplatform systems integration to network security and technical support. Visit our website at www.rht.com.

RHT: Data Modeler: Decipherer of Data

by Robert Half Technology
July 14, 2014

Starting pay for data modelers is expected to increase 5.8 percent this year.
Today’s data-driven businesses need experts to help them access and transform raw data into information that management can interpret and use for decision making.
According to Dave Kaplan, branch manager for Robert Half Technology in Charlotte, N.C., data modelers are “lynchpins” in the process of gathering, analyzing and organizing data so it can be easily understood by nontechnical users.

Kaplan explains that the data modeler — who is part traditional data analyst, part business analyst and part problem-solving guru — is needed not only to help the organization determine what type of data is required for a particular business process or objective, but also to figure out from where and how the data can be sourced, and the best way to present that information.

Data modeler salary increasing in 2014
The translation of business requirements and objectives into data models is crucial for growing businesses, which means skilled data modelers are in demand. Robert Half Technology’s 2014 Salary Guide reports that the average starting data modeler salary in the United States* is expected to increase 5.8 percent this year, with numbers ranging from $97,250 to $134,250. (Use the Robert Half Technology Salary Calculator to find specific salary information for your city.)

What it takes to be a data modeler
Employers look for relevant data management experience when assessing candidates for data modeler jobs. A data modeler also should possess the following skills, attributes and experience:

• Excellent data analysis and problem-solving skills
• A bachelor’s degree in computer science, IT or mathematics
• Familiarity with data modeling tools and methodologies
• Knowledge of database system applications, stored procedures and data warehousing

As a data modeler, your main objective is to convert a mass of information into proper computer coding and translate that information into a form that is ultimately useful in realizing business goals. For example, you may be tasked with reducing the redundancy of data within a database or improving the way information moves from one system to another.

Data modelers also should be able to understand different operating lines in the business and identify the best way to help those functions meet their goals through better access to relevant data. Analyzing organizational data requirements and creating logical and physical models and workflows based on that data are key job requirements.

Because data modelers need to interact directly with a wide range of functions in the business,communication skills and the ability to work as part of a team are essential. In the data modeler role, you’ll be in charge of interviewing key project stakeholders and later documenting your findings, as well as making detailed recommendations based on internal and external client needs. You’ll also work side-by-side with developers, database administrators and reporting teams to ensure consistency between standard reporting and reporting that’s not already pre-defined in a production environment. Likewise, you’ll be addressing data quality issues with clients and management.
Aspiring data modelers: show your love for numbers

When it comes to making an impression with a potential employer, the more exposure you have to different data modeling tools — such as the CA ERwin Data Modeler or the IBM InfoSphere Data Architect — and methodologies — like View Integration or top-down logical data models — the better, according to Kaplan.

“Don’t be shy about your love of numbers,” he says. “Showing off your background in mathematics and related technical experience is always a plus.”

Look to Robert Half Technology’s latest Salary Guide for job descriptions and starting salaries for a wide range of IT jobs — including data modeler. Explore the Robert Half Technology Tech Blog to learn about hiring trends and skills requirements for other data-focused professions, like data architect and data warehouse analyst.

*The starting data modeler salary in Canada is projected to increase 5.6 percent in 2014, ranging from $85,750 to $116,750. Figures are in Canadian dollars.



— Robert Half Technology

With more than 100 locations worldwide, Robert Half Technology is a leading provider of technology professionals for initiatives ranging from web development and multiplatform systems integration to network security and technical support. Visit our website at www.rht.com.