GIS Location Analytic turns Big Data to Information & to Understand

My first encounter with Big Data was during my university studio project – part of my course work as Town Planning student in University Technology Malaysia (UTM). Our lecturer will divide the class into 2 teams and we were given an assignment to prepare a regional development plan within 1 semester. We started our assignment by gathering policy, guideline, zoning data from local authority, state agencies, environmental department, irrigation and drainage department, etc.  At the end of the assignment, we will present our proposed regional development plan to our fellow course-mates and lecturers and substantiate our proposal with rationale backed by facts and figures.

One of the usual comments given by our lecturers that I still remember is – Our group has gathered loads of data but failed to turn them into information to support our rationale/analysis/proposal”. On hindsight this can be due to many reasons, among others not enough scientific methodology was used to process the data, we were not effective in communicating our findings, or lacked of visual aids to integrate various data sources into useful and easy to understand information for our audiences.

Have you had similar experience in your recent presentation to your bosses/stakeholders?

Hence, challenges associated with big data are not new. It is about how to effectively turn massive amount of data into useful information that will help you make decision, whenever you need it and in format usable for you. Else the data still remain as data and it wouldn’t help you to be more informed.

Richard Leadbeater, Esri State Government Industry Manager once wrote “Big Data: The Three Vs and the Missing L“ – there is much attention applied to the three Vs of big data – Volume, Velocity and Variety of data. GIS provides the foundation for information integration by assembling massive amounts of data – often, types of data that are unrelated except for their geographic Location. He opines GIS map provides better understanding and reference than an informative spread sheet.

Nearer to home, Datuk Prof. Sr Dr. Abdul Kadir Bin Taib (Director General of Department of Survey and Mapping Malaysia, JUPEM) once said this in his conference speech, he was appealing to all attended to not just manage geospatial data, but manage all our data geospatially. According to him, managing our data geospatially will allow us to be more informed and our data is more accessible in the form of useful information, where and when we need it, that will help enhance our understanding by leveraging on the location analytic capability of GIS technology.

Indeed, location analytic helps many enterprises to take advantage of this processed information to next level and reveal greater insights and understandings. GIS is the most important element of location analytics. It enables users to turn data into information and add in 2D/3D location dimension, as well as 4^th dimension (time) to produce useful insights/pattern (distribution), co-relations, hidden in between rows and rows of textual/database records. If a picture can tell a thousand words, how will you describe an informative/interactive/analytic map that can generate endless scenarios and analysis supporting your decision-making?

Location analytic is not just meant for business community but any organizations that would like to drive performances or Key Performance Indicators (KPI), be it in utilities, public sectors, education, finance/insurance, transportation, retail, etc. To read more about location analytics, I have uploaded a good white paper from Esri explaining Location Analytics for Business Intelligence in my blog Library.

Importance of Metadata (Data Label)

Metadata Publication from MaCGDI

Normally, there are few things that I will check before buying an item. Among others the brand name, expiry date, nutrition content, etc. these are important information for consumer and we are grateful most of the consumer friendly manufacturers oblige with the labelling guideline & regulation.

In most occasions, the labelling standard is strictly regulated and the consumers do not need to second guess the contents because someone else has done the quality control and inspection for us. For example, we do not need to worry the fuel from our petrol stations will be damaging to our car engine, as long as we do not mixed up diesel and petrol fuels.

I can’t imagine what will happened if our fuels were without proper ISO standard or labelling (diesel, RON 95, RON 97, etc), without knowing the gasoline content, without knowing the quality rating, etc. how are you going to know either you are choosing the right fuel type for your car?

Similar in geospatial industry, we need proper labelling for our GIS data. This is one of the important milestone in national geospatial development and MaCGDI (Malaysian Centre for Geospatial Data Infrastructure) is actively promoting the importance of metadata among GIS users.

The U.S. FGDC (Federal Geographic Data Committee) defines geospatial metadata as: “A metadata record is a file of information, usually presented as an XML document, which captures the basic characteristics of a data or information resource. It represents the who, what, when, where, why and how of the resource. Geospatial metadata are used to document geographic digital resources such as Geographic Information System (GIS) files, geospatial databases, and earth imagery. A geospatial metadata record includes core library catalog elements such as Title, Abstract, and Publication Data; geographic elements such as Geographic Extent and Projection Information; and database elements such as Attribute Label Definitions and Attribute Domain Values.“

Unless you are operating in isolation, else in most instances you will need to exchange data with someone else. In order for your data to be discoverable and meaningful to your potential users, your data need to have few simple labels among other like title, content description, keyword, date of creation, data owner, data limitation etc. These will provide your users with sufficient details to understand the source of the data that he or she is going to use with greater certainty.

Simply put, Metadata is label for your data. Without a proper descriptive label, your data (product) will not be meaningful to your users. Even in small operation, as a manager it is always a good practice to instil the discipline of maintaining your metadata for easy retrieval and reference in future.

Standard GIS Basemap – OneMap Malaysia (1Malaysia Map)

Sample information product from OneMap Singapore

There’s a saying that goes: “The way to man’s heart is through his stomach.” Because man loves food and cooking for a man is a sure way to win his affection.

I have another saying that “the way towards national GIS is through standardized basemap.” Simply because GIS users need map data and maintaining a standard basemap for every GIS users is definitely a sure way to ensure data interoperability.

However, what is a national basemap? While the map contents or layers can be the more the merrier, but there are some basic elements that needs to be included as foundation of basemap. While some might need specific data pertaining to their area of interest, most GIS users simply need a good street map that contains the following basic layers :-

• Street network & hub with name i.e roads, railway, LRT, stations, airports, port, etc.
• Land use area with name, building outline, parcel layout, address
• Location and name for important places or point of interest (POI)
• Administrative boundaries with names – national, state, district, mukim, postcode, etc
• Hydrography and other natural features with name i.e. rivers, lakes, etc.
• Imagery i.e. satellite images, aerial photos, etc if any

These basic layers can form a pretty good basemap similar to street maps or city maps that can support most GIS activities. Other Agencies can add more textual data or attributes to these geometry (in the form of points, line and polygon) and make them more meaningful for their own purpose. The main point is as long as everyone is using the same basemap as their references, then we are on the right track towards national GIS.

This is pivotal because GIS is model after the real world objects and organize them into layers of information. The ability to overlay the information correctly is fundamental in data sharing and collaboration. In other words, it is useless to have different layers of information from Ministry of Health (MoH), Ministry of Education (MoE), and Public Work Department (JKR), only to find out that each and every one is maintaining their own version of basemap with different level of details, degree of accuracy and map scale. Hence the location of hospital, schools and road networks might appear correctly in respective GIS system but not the case when other users wish to overlay all these information for other planning purpose.

Imagine different parts of a car engine were manufactured by different companies without adhering to a common design layout or framework. Once the separate engine parts are finally ready, it might not fit into the car chassis compartment because different manufacturers are using different type of bolts and nuts, materials etc that suits them best without the bigger picture in mind.

Singapore Land Authority (SLA) started a very good initiative few years back and named it OneMap Singapore. You can take a look at their website at http://www.onemap.sg/index.html

OneMap Singapore is a web-based integrated map system for government agencies in Singapore to deliver location-based services and information for the general public. Many government agencies collaborated and continue contributing information to enrich the data content. Among the services available on this portal include: find an address; get travel directions; find a school to enrol your child; query land ownership and land lot information; check on historical property prices; find properties for lease; find free public wireless hotspots, etc

SLA as land base map custodian in Singapore makes their map available for all government and private organizations. In return, public and private organization will make use of this standardize basemap to add in their operational layers, i.e. Ministry of Education will link the basemap to their website and add in more educational related information, Ministry of Heath will make use of the basemap to add in more healthcare information, other agencies will make use of the basemap to add in their coverage area, zoning, etc. At the end of the day, even though SLA offers their data for free, but they emerge as the biggest winner because they are managing a critical national geospatial assets that all agencies will depend on. They are holding the car chassis design framework layout which enable all other departments to simultaneously but independently builds their respective engine blocks that will eventually fit well into national building.

We need to have our OneMap Malaysia (or 1Malaysia Map) as foundation for our national building too.

Commercial Off-the-Shelf (COTS) Approach – More configuration less customization

Esri is actively promoting COTS approach in Enterprise GIS implementation.

Let’s recall for a moment, after we purchased and Installed Microsoft Office, how many of us will consider customise the Microsoft Word, or Powerpoint application to follow our way of working? Or imagine that every user tries to customize the Microsoft Office interface and make it looks unique by developing new add-in using self-invented API or object library.

I can hear you screaming out loud. “Noooo, that’s ridiculous!”

“Why do you want to do that?” You continue asking.

“You can make use of Microsoft word and Microsoft Powerpoint out of the box after installation, why bother to change it? Besides, if you customized it, my life will turn upside down if all my colleagues’ Microsoft application looks different from each other. Worst when Microsoft rollout new release (which they will), the self invented add-in will become unsupported or I wouldn’t able to upgrade to the latest version and enjoying all the new features of Microsoft because too much add-in were customized.” I can hear you continue to lecture me.

Ok. ok. I hear you loud and clear and it does makes a lot of sense. But to my amazement, we are doing exactly the opposite when comes to implementing GIS project. We still tend to customize every application to make it only uniquely ours, regardless the risk that one day the developer that developed the application might no longer with us. Or the tendency to reinvent the wheel by committing precious resources to do R&D or continue maintaining the aging apps.

We have too many experiences that heavily customized GIS systems cannot be easily scaled or extended because there is inadequate documentation to perform transfer of technology, or the technology is no longer supported; or the resources that has the specific programming skill sets are no longer available.

The following excerpts are extracted from a great whitepaper published by Esri titled “Taking a COTS-based approach to implementing enterprise GIS“. I encourage you to download and read the full text.

For those of you that would like to read only the executive summary of the white paper, I have extracted the following short paragraph for your reading pleasure.

The fundamental premise of a COTS approach is to exploit all the power and functionality the commercial software has to offer, minimizing custom development.

Implementing the COTS approach requires fresh thinking, leadership, and agility. This can be challenging in environments where there is a history of procurements to build custom systems. One key difference is that the COTS approach is predicated on a focus on business goals instead of a list of detailed feature functions. This allows the organization to select the COTS solution that best meets business goals and to implement the COTS solution rapidly with the grain for best results.

This approach asks users to consider new business processes. COTS solutions provide new workflow capabilities based on industry best practices, which are often better than legacy business processes. Traditional systems procurements often fall into the trap of re-creating old workflows out of new software, usually because people and organizations resist change. It takes leadership, communication, and follow-through to overcome the tendency to stick with the familiar (and usually less efficient) legacy workflows.

An additional trap that users must avoid is the temptation to customize rather than configure. Even when a system has been proclaimed a COTS implementation, the best intentions of many people often push systems toward customization. The trade-offs of custom development must be kept in mind during the decision-making process to avoid moving a COTS-based system into a custom or component-based implementation. Many heavily customized COTS systems cannot be easily scaled or extended because there is inadequate design documentation; the technology on which it is based is no longer supported; resources with needed skill sets are no longer available; or the COTS, as opposed to custom, modules are not clearly delineated, creating dependencies between components that minimize the potential for reuse.

Single Family Product – One Brand or One Technology?

Over a tea chat, a friend of mine with IT background was explaining about challenges associated with merger and acquisition (M&A) involving software technology companies. According to him, while a conglomerate might able to acquire companies within months and rebrand the products into one family, but the technological integration between the software products previously developed on totally separate architecture, platform and maturity will be extremely challenging and will take years if it is possible at all.

With his many years of experience in IT industry, he must have encountered many legacies integration challenges & issues related to M&A activities. When company H acquires another company and later aggressively rebranding the products, behind the scene, it is not uncommon to hear it take lots of effort to standardize the different accounting, CRM, and sales systems into a common corporate system. In many cases, it is probably easier to retire a system or rewrite another instead of trying to integrate these different legacies systems. That’s how challenges it is for in-house accounting/CRM system, and the same will goes to different GIS software/ modules/ patents inherited from the company being acquired, which was developed on totally different platform. The fact is the integration will remain challenging regardless of before or after the acquisition transaction. Unfortunately, the technological challenges will not fade away simply because the marketer has done a great job to blur the facts and successfully rebrand the products under a single family product.

I must thank him for sharing this insight with me and make me wiser in understanding what’s more important to consider when implementing IT projects involving multiple software modules/ solutions. There are huge differences between common technology platform family and ownership family when comes to software solutions. Unmistakably, it is important to go for common technology platform family rather than ownership family if one wants to have common operating platform within the organization.

Simply because it is definitely going to be more seamless, safer and easier if the software system are developed on same technology platform regardless either it is owned by the same owner. Take for example the ArcGIS system, it is integrable seamlessly with Exelis VIS ENVI systems, BAE SOCET, Telvent ArcFM. All these software are owned by different companies (different brands) but can be seamlessly integrated under 1 common technology ArcGIS environment. It simply means less integration hassle and headache to make these systems works seamlessly within the new post-acquisition environment.

For GIS manager, it make sense to take a deeper dive and understand lots more from IT perspective before believing the marketing gimmicks. If your technical staff is not taking technology platform integration issues seriously or told you not to concern about it. Be worry. Be very worry. Because he/she is either not technically competent or he doesn’t care about the consequences. Either case, you as the manager in-charge are going to be in deep trouble. Bear in mind, while technical personnel always can justifying the unknown but that’s not going to save your day as manager/director of the company.

Lesson learned – be sceptical when someone claims they have single family product especially the products were acquired through series of M&A activities, which might not be technological integrable at all.

So there you go, “One Family Product” can be interpreted from marketing or technological integration perspective.

Thanks buddy, I will go for “One technology platform with independent brands” instead of “One brand with independent technology platforms”.