Persistent Systems Blog

 By: Shivesh Vishwanathan, Senior Architect, Persistent Systems 
       

The recent surge in mobile technologies is changing the way we do business and communicate internally in our organizations. While businesses continue to struggle in understanding the implications and benefits of increasing mobile usage within the organization, the shift is part of a larger trend - a trend of increasing the level of employee involvement from the bottom up. Today, corporations need its employees to take a more proactive role -for example, they want their line employees to relay information back in a meaningful way, engineers to share best practices, and managers to work closer together in cross-functional initiatives, etc 

Traditionally, mobility has been embraced within enterprises because of its two key attributes that increase employee bottom-up involvement : Reach and Response. Reach enables companies to reach out to employees like never before. It enables communication with the sales team in the field with important information. Response enables  employees  to dynamically respond to developing situations. The combination of the two increased the velocity at which business is conducted.

The Third Leg of Mobility

“If Facebook were built today, it would be a mobile app” – James Pearce, Head of Mobile Developer Relations for Facebook

 Similar to Reach and Response, Relationships is also best served mobile. People are increasingly interacting with social media through their mobile phones and tablets. Some commentators have indeed gone as far as to say that social is the killer app on mobile or it is the killer app for convergence. Facebook ended 2011 with 432 million mobile monthly active users(MAUs), which is over 50% of its total user base of 845 million monthly active users. This is a whopping 76% year-over-year growth in the mobile market.  13.4% of the 432 million users (or 58 million) access the site only from their mobile device.

It is not very surprising if you think about it. Social features are more informal, and things like device notifications are much more suited for collaborative work than say, emails. Mobile phones provide the delicate balance between information sharing and intrusiveness that make them the right channel for social communications. Facebook’s recent acquisition of Instagram is also to a large part a testament to the advent of mobile-centricity in social networking. As more and more mobile-centric features such as photos, voice and others become mainstream, mobile and social will come closer and closer together.

Mobility has had a somewhat precarious existence within enterprises, and has been limited to only critical functions that could exploit its Reach and Response abilities (evidence being the popularity of RIM’s Blackberry devices and services). To go well beyond these basic benefits, enterprises need to use mobility to build strong Relationships amongst users. Using mobility to expand their collaboration portals – like having employee profiles, enabling following, alerts and sharing – would be a good start. Mobility, for its part, just got a third leg to stand on its own within the Enterprise.

Organizations are realizing the business importance of deploying Collaboration tools. But, just realizing the importance of these tools is not enough. What is of importance is executing the collaboration strategy. How can this be done? The first step is to develop the enterprise collaboration architecture and the second to approach the technology decision.

While the first whitepaper spoke about how organizations need to develop a business driven enterprise collaboration strategy and approach, the second explains how collaboration architecture can be developed and the different obstacles you would encounter as you develop this strategy.

Enterprise Collaboration Architecture is all about how collaboration tools enable your employees to achieve their business objectives; how these tools relate to the other technology systems and most importantly how these tools help you achieve your organization’s business goals.

Further, in this whitepaper we talk about organizational and technological considerations while deploying the collaboration platform. How Change Management lays at the core of the Collaboration platform, overcoming barriers to user adoption, the problems of Governance and we discuss topics such as Scalability, Usability, Integration, Legal Compliance and Security.

The third whitepaper, which would be the final one in our Collaboration series, will highlight a select group of enterprise collaboration platforms and offer a step-by-step guide for choosing the right collaboration technology. So, keep watching this space for the whitepaper which will walk you through the process of evaluating collaboration platforms based on your business case, organizational considerations, and technology considerations.

To download this FREE whitepaper, click here

 By: Shreekanth Joshi, AVP -Cloud Computing, Persistent Systems 
       Satish Nikam, Architect, Persistent Systems

The demand for Java-based applications on Windows Azure is increasing as customers are realizing openness of Windows Azure that can provide scalability and high availability to their Java applications .We get a lot of questions about how to design various project components to manage single-tenant and multi-tenant applications, how to integrate project components with Windows Azure services. The challenge often faced by our customers while learning Windows Azure is that there is only a limited amount of informative articles and code samples that cover platforms other than NET.

Windows Azure is often perceived as being a .NET Cloud Platform, which isn’t true. This misconception is based on demos and how-to blogs that are written around Microsoft Visual Studio. As it turns out, Windows Azure provides virtual machines that are either Windows Server 2008 SP2 or Windows Server 2008 R2, meaning that most of the Windows-based executable or scripts can be run on Windows Azure.

To increase awareness about the openness of the Windows Azure, we are releasing an Open Source Project, CloudNinja for Java , a reference application to illustrate how to build a multi-tenant application for Windows Azure.

CloudNinja for Java

CloudNinja for Java is a multi-tenant reference application designed to help the Java community develop Java applications for Windows Azure. In addition, our application also implements best practices and design patterns required for Java multi-tenant applications.

 The CloudNinja for Java application encompasses following features and functionalities:

• Tenant on-boarding
• Tenant-level customization (for example, managing logos)
• Per-tenant data isolation
• Per-tenant metering
• Providing support for log-in via different identity providers (for example, Yahoo!, Google, Windows Live ID)
• General purpose task scheduler

This application is built on several common OSS libraries, such as:

• Spring for the MVC UI
• JDBC and Hibernate for data access and object mapping
• Log4j
•  jqPlot

The project runs in Windows Azure and was developed entirely using Windows Azure Plugin for Eclipse with Java. Here is the illustration that depicts the architecture of CloudNinja.

We utilized various Windows Azure services in the development and deployment of CloudNinja for Java. Some of the most important Windows Azure services used in this Java application are:

·         Access Control Service (ACS)

·         Windows Azure Storage

·         SQL Azure Database

·         Windows Azure Service Management

We believe that the CloudNinja for Java application will be beneficial to the Java community and encourage the Java developers to create their own applications for Windows Azure.

Interesting Features used in CloudNinja

Here are some featureres we have implemented in CloudNinja for Java.

1. Use Access Control Service (ACS) for aunthentication using multiple Web Identity Providers

Windows Azure ACS Service has out-of-the-box support for popular web identity providers including Windows Live ID, Google, Yahoo!, and Facebook. We used ACS to enable authentication with these Web Identity providers.

2. Use REST APIs for Windows Azure ACS Management

Some Windows Azure features are not yet provided in the Windows Azure SDK for Java. For example, we wanted to manage ACS for creating relying parties from CloudNinja for Java application. As Windows Azure SDK for Java does not have any API for ACS management, we used REST APIs for ACS Management using a REST library (Restlet). The same technique could be used for any other REST-based feature in Windows Azure.

3. Monitor folders using Windows Azure Diagnostics and push the folder data to Windows Azure Blob Storage

Windows Azure Diagnostics is typically used to collect performance counters like processor idle time and available memory. In addition, it can also monitor a folder for any changes and push the content of the folder to a blob container periodically. We used this functionality to monitor the Tomcat access logs and process them to prepare usage statistics.

4.  Reducing the Deployment Size

Windows Azure plugin for Eclipse bundles Java Web Server (such as Apache Tomcat) and JRE into the package to be deployed on Windows Azure. However adding Tomcat and the JRE adds about 70 MB to the package size resulting into longer time to upload the deployment package to Windows Azure. In addition, if we ever want to update the version of Tomcat or the JRE, we have to redeploy the entire package.

To get rid of the above mentioned issues, we stored them in Windows Azure Blob storage. We modified the startup script to download Tomcat/JRE from the Blob storage and then launch Tomcat. For upgrading the deployment to use the latest Tomcat and JRE, we needed to copy new versions to the Blob storage and using Windows Azure portal, we reimaged all virtual machines instances that host our application.

4. Metering 

In a typical multi-tenant application, various application resources are shared across all tenants. There are many business models that service provider can use to charge each tenant. We showcased one model which can be used to charge tenant based on their actual usage of each resource (in our case, Windows Azure Storage). For capturing the storage usage for each tenant, we enabled Storage Analytics logging for the Windows Azure Blob storage. This creates the Storage Analytics logs, containing the information about the operations performed on the Windows Azure Blob storage service. Storage Analytics logs also contain the information about the successful and failed requests made to this service. Storage Analytics is a great Windows Azure feature that was especially helpful for CloudNinja for Java, being a multi-tenant application for which we wanted to capture the tenant-level storage usage.

Blog Series on Windows Azure for Java

Watch out this space for in-depth series of blog posts covering the various aspects of Windows Azure for Java, such as:

• Usage of the Windows Azure SDK for Java
• Accessing the Windows Azure API through REST-based calls
• Integrating Windows Azure Access Control Services for authentication and authorization

Apart from this we would also be documenting some interesting observations while implementing Java applications for Windows Azure. So stay tuned for regular and informative posts that will help you learn about Windows Azure for Java.

We recently conducted a joint webinar with IBM - Big Data on BigInsights on 23rd Feb 2012. In the Q&A session, there were many questions raised by the attendees and we thought it would be a good idea to answer them on the blog and share it with everyone.

If you want to watch the webinar, here’s the link to the recording

Q1. On BigInsights, you were mentioning large scale indexing, can you expand more on it? How are these indexes created or maintained? Do I need install additional software?

BigInsights comes with a module called BigIndex, that allows you to build an index unstructured data as well as query the data once the index is built.

BigIndex is built on top of Apache lucene.

Q2. Can you please elaborate Hadoop use in Persistent product ?

The email analytics solution is built on BigInsights (which contains apache hadoop). The solution consists of an email connector to directly load data from an email server, a text analytics pipeline that analyzes the email data and finally a connector/data loader that transfers the results in either an olap solution or a search index for easy retrieval.

Q3. Will you provide some session on infosphere Biginsight? 

Additional information about IBM BigInsights is available at  http://www-01.ibm.com/support/docview.wss?uid=swg27023211 

 Q4. What is the role of HIVE in big data analysis? Does the velocity of data have a direct impact on big data systems?

 Hive provides a SQL like interface on the data that is stored on Hadoop, in that way it provides easy access to the data in Hadoop. Please note that SQL support is limited and only the basic SQL commands work.

Actually, big data systems (hadoop in particular) are designed to handle large amount of inserts, and in that sense they are superior to some of the other analytics systems. 

Q5. Does the solution/product allow data and/or metadata to be exchanged between the various systems? Or does the customer have to do that manually?

Not sure about which system, but currently Hadoop does not contain any high level meta data management software that is available, obviously it does manage meta data about the data blocks, etc. But typically it will not be relevant in working with other system’s context.

 Q6. Hive itself provide SQL interface then why to use your system?

The SQL supported in Hive is very minimal – only basic operators are supported. also SQL by nature is declarative and will making writing advanced analytics difficult. Finally the performance of Hive is not very optimal.

Q7. Can you walk us through how the user interacts with the Bigdata platform?  I understand the process of aggregating unstructured data. I don’t understand the process of asking questions to the system. 

There are several ways of querying the hadoop platform – starting from difficult to easy

1.      Writing Java code on top of the map/reduce Java apis, this will require you to write code in Java and implement your logic (queries) by extending the map/reduce methods, most difficult but you have full control and can optimize it the best.

2.      Write queries using higher level languages like JAQl/PIG: these are higher level languages that will convert the code your write in them in map/reduce functions and execute on the platform. This  frees you up from thinking map/reduce but you have less control and in some cases the map/reduce code generated may not be most efficient.

3.      Use Apache Hive and implement your logic in a series of SQL functions, similar to JAQL/PIG in spirit, with an exception that SQL is well known (no need to learn new language) but SQL support is limited also the indexing support is getting implemented. You will have to spend some time setting this up.

4.      Use higher level tools like IBM Bigsheets, or an offering from datameer, that give you a visual environment (like a spread sheet) to visualize and process your data. Obviously you are further removed from the map/reduce layer on Hadoop so performance challenges are harder to debug, functionality is limited to what is available in the products, yes UDFs of some kind are available.

Essentially you will have to choose the method that works best for you to ask questions.  

Q8. Is Open Data (Public data in US and Europe and other places) a new opportunity for democratic BIGInsights ? Do you have experienced use cases with it ?

Yes, this is definitely a new opportunity and with Bigdata platform being touted as an enterprise data repository, I definitely think that it would be an ideal place to integrate it with public datasets.

Usecases: There are several available, I can think of several, specific on is available from IBM. (http://www-03.ibm.com/press/us/en/pressrelease/35737.wss) This describes how Vestas used weather/tidal data available for improving their turbine placement, and optimal energy output problem. 

 Q9. Were the use cases presented here solved by IBM and Persistent Systems collaboration ?

The email analytics use case was built by Persistent, we have collaborated with IBM on some of the use cases mentioned in the presentation.

 Q10. What is there for the business users?

Please review the earlier answer about interacting with the hadoop system.

The tooling around Bigdata is still evolving so it is not very straightforward for business users to interact with bigdata systems, however this is one area where lot of vendors are focusing so things are bound to improve.

Simultaneously, I think we will have vertical specific Bigdata offerings, that will be targeted for the business users of that domain, we should see a lot of those in next 6 months to an year.

Q11. Can you share some big data architectures slides

Persistent’s Email Analytics Architecture

Q12. How does rational tooling support development of BigData applications?

Internally from IBM perspective what they are doing is reaching out to the various plans both within software groups and obviously within STG which is our hardware division to make sure interoperability. Different products are available today we have got interoperability with DB2 and some other products would be coming down the row like SPSS and COGNOS as we move forward. So that’s basically where we are again is different products in tooling is available when outsource make sure that you have clear understanding of what specific products work in this portfolio. But at end of the day again general statement and direction without preannouncing that this stuff is available or intended to make it again reaching so that we got that interoperability.

(Answered by: Vish Vishwanath - Senior Vice President, BI & Analytics, Persistent Systems

Anand Ghalsasi - Associate Vice President, Sales, Persistent Systems

Mukund Deshpande - Associate Vice President, Operations, BI & Analytics, Persistent Systems)

By: Mukund Deshpande, Associate Vice President - Operations, BI and Analytics, Persistent Systems

      Kartik Vyas, Lead Business Analyst, BI and Analytics, Persistent Systems

While everyone seems to be convinced about Big Data being the next big thing in technology, there seems to be a certain amount of uncertainty in regards to the ways in which enterprises can leverage Big Data. The question is whether or not Big Data deployment is worth it and if yes, how much?

Traditional financial measures such as Internal Rate of Return (IRR), Net Present Value (NPV), Payback Period or Return on Investment (ROI), can be applied to Big Data projects. Typically cost benefit analysis takes into consideration the monetized value of the proposed benefit and the cost incurred through implementation. Although this applies in most Business Intelligence/Data warehousing projects, it doesn’t necessarily hold true for the Big Data projects. In order to fully evaluate any Big Data project from a financial standpoint, organizations have to take into consideration the non-quantifiable benefits of Big Data projects as well.

Listed below are a few non-measurable factors which organizations tend to forget while calculating returns on the Big Data projects.

1.     Do more Analysis with all of your data and less investment: With Big Data technologies, multiple business scenarios could be run on all the complete data set, rather than a traditional sampling. With no statistical errors, your insights will be accurate, this cannot be analyzed in pure ROI terms

2.     Enhanced incremental ROI: Cost of implementing new insights is significantly less than traditional BI techniques for two reasons:

a)     The price of traditional BI solutions is linearly dependent on the size of the data. For incremental insights all you need is a few more machines

b)     No specialized hardware is required for Big Data projects, commodity hardware is more than sufficient, thus a low cost of new insights

3.     Scalability: Big Data technologies offer effortless scalability as your data infrastructure could be readily scaled to handle higher data loads without altering your current data setup.

4.     Ease of adjustment to changing data requirements:  Traditional BI implementations tend to be non-effective the moment data points or structure change. Fortunately, Big Data technologies don’t have this constraint; any change in data pointers, schemas, or volume of data would not require any modifications to your data systems. If time is money, this flexibility should be considered while monetizing the benefits for a cost versus benefit analysis.

5.     The chance of hitting a jackpot:  Big Data projects are aimed at deriving insights from data pointers that could not be previously.  There is always a probability of inadvertently discovering critical insights which may be worth much more than the initial consideration.

Obviously, strategic as well as operational considerations also play a big part in deciding whether or not to go for Big Data. But these broad (or should we say big) insights need to be considered while analyzing any Big Data project from a financial standpoint.

 For information on your Big Data queries, write to us at  BigData@persistent.co.in  or visit our website

 By: Mukund Deshpande, Associate Vice President - Operations, BI and Analytics, Persistent Systems
        Darshana Bhayani, Delivery Manager, BI and Analytics, Persistent Systems

Discovering trends and patterns in seemingly uncorrelated data can equip businesses with the competitive edge needed to add value and differentiate an enterprise. As the amount of data increases, there is a growing need to quickly analyze varied and vast data for meaningful insights.  

There is plethora of information in various forms such as web logs, email and more. In addition to its size, the unstructured nature of the data creates problems. With Big Data technologies, unstructured data generated from a various sources can now be analyzed. Big Data technologies tackle issues related to the variety, volume, and velocity of data. The challenge lies in the necessary rapid development of tools capable analyzing data as per business requirements.  In terms of analytics, business requirements vary with the industry, but there are specific set of functions and algorithms which are primarily required to build any Big Data solution.

Having worked on several Big Data projects area across diverse applications such as search engines, recommendation engines, and email analytics, we at Persistent Systems found that these varied applications have common underlying structure. Social networks, transport networks, and the web are essentially some sort of graphs. There is an ardent need for functions on these graphs; finding shortest distance between two cities or finding how people are linked in social networks. Similarly, email analytics, sentiment analysis, or search engines need their text to be extracted. Instead of reinventing the wheel every time, we thought it would be more useful to have a ready made library of functions, which can be utilized for Big Data application development across diverse domains. Hence, we built the Persistent Big Data Analytics Library (Pebal)

Persistent Big Data Analytics Library is a library of commonly required functions which can be utilized to build Hadoop based Big Data solutions. Several functions have been identified across five major algorithm areas i.e. Graphs, Sets, Indexing, Text Analytics and Web Analytics.

For instance, typical requirements of text analytics on email data are masking personal information or extracting entities from documents. Pebal provides high performance, easy to use algorithms for several commonly required functionalities.  Like an STL library for C++ development, Pebal functions significantly to reduce time for development and deployment of solutions in Big Data world.  Pebal functions are generic, easy to learn and use, and have been tested on large data sets. These functions have schema on read paradigm followed by Hadoop, are schema agnostic, and use JSON formats for data and schema.

For more information please visit Pebal on our website keep watching the Big Data category on Persistent Systems blog for more insights on Hadoop and Big Data.

By: Mukund Deshpande, Associate Vice President - Operations, BI and Analytics, Persistent Systems

      Kartik Vyas, Lead Business Analyst, BI and Analytics, Persistent Systems

Everyone is talking about Big Data and how it is slated to be the next big thing. However, a novice is likely to be intimidated by the sheer enormity of the buzz to dare ask, “What is Big Data?” So here is an honest and simple definition of Big Data.

Imagine if a survey of one hundred people, asked the question, How big does a house need to be to qualify as a big house?  The answers would range from 1000 sq² to possibly 50,000 sq². So what qualifies a house as a big house? Maybe I would define a big house as one which is big enough to accommodate each member of my family with their own bedroom, in addition to a guest room. Sounds logical? If yes, then that’s how Big Data is defined.

Big Data is data which is too big to fit into the current data devices of an organization. For example, a startup with just a VM as their database, even 10 terabytes could be considered Big Data.  For a big corporation, even Petabytes of data may not qualify as Big Data. The definition of Big Data depends on your current capacity.

There is more to Big Data than just size.  Big Data can be used to derive intelligent insights and to create sustainable and competitive advantages for corporations.  But first, why do enterprises need Big Data?

As an enterprise, you need Big Data technologies if your current analysis needs are constrained by one of the three V’s of data –Volume, Variety and Velocity. For example, you would probably need Big Data Technologies if:

• your dataset to be analyzed is too voluminous to be handled by your current data infrastructure
• there are too many data sources from which you need to pull data or your data is in various formats
• you need quicker insights and the velocity at which your data changes is too fast for your current systems to handle

A lot of enterprises are leveraging Big Data technologies to derive deeper insights from a continuously expanding set of data. Many of these insights are obtained real time at a fraction of the cost that enterprises would normally make had they gone the more traditional BI route. Watch this space for other stories concerning Big Data technologies.

Location Based Services (LBS) is one of the hottest trends in mobile services today. Location services well accepted by users, range from core location services like emergency services Enhanced-911 (E911) to various commercial services like navigation, fleet tracking. Taking it further, market expects to see more traction in the areas of location mesh-up with services like social networking, mobile advertising and enterprise collaboration.

Further, the Mobile cellular ecosystem is continuously evolving to meet the needs of mobile users. One of the important requirements is having high bandwidth data network accessible on the move. This can enable host of applications like video calling, conferencing, etc. To address these needs, wireless networks are continuously evolving, with Long Term Evolution (LTE) being one of the most promising fourth generation (4G) technologies. However, the cost and complexity involved with new network deployment poses challenges to carriers on how to effectively ensure Return on Investment (ROI). Launching popular services such as various LBS helps them many times in improving the ROI.

Location technology being an integral part of wireless network, with network evaluation it is evident that location technology would also require corresponding evolution. To address this, various standards bodies work towards different solutions for meeting different market needs and no single solution can suffice for all the market requirements.  This poses challenges to carriers, their infrastructure vendors and even handset vendors in selecting the appropriate technology to meet their specific market needs.

At Persistent, we have developed significant expertise in location technology and applications by working with different infrastructure vendors, Tier 1 carriers and applications solution providers.  Utilizing this experience, we have recently written a white paper on Delivering Location in LTE Networks which helps provide some direction on how to deliver location in LTE network. It talks about available technology options and how these diverse choices can meet the varying market needs of different vendors and carriers.

We welcome and greatly appreciate your thoughts and comments on the whitepaper.

Download this FREE whitepaper now

By : Shreekanth Joshi, AVP -Cloud Computing, Persistent Systems 

       Vidyut Kumar, Senior Technical Specialist - Cloud, Persistent Systems

At Persistent, we have been finding a common trend amongst our customers: enterprises  are challenged with systematically identifying, categorizing and assessing on-premise applications for Cloud migration.  With the seemingly daunting and difficult task of Cloud migration, business leaders feel the need to understand migration criteria along with an analysis of the ROI and TCO perspective.  

Cloud migration is similar to a backpacking trip: without a map, a proper plan, and the correct tools, it is easy for a business to get lost.  Given the limited luggage space, we can bring only what is necessary, likewise understanding the reasoning and potential benefits of moving applications to the Cloud, from both a business and technical standpoint is critical for success. 

In the current landscape, most of the assessment tools are primitive; data collection consists of interviewing IT and business users.  Assessments of inaccurate data are carried out manually and lack clarity with respect to understanding the actual workloads associated with an application. 

What enterprises need is a tool that can provide a clear analysis of the assessment and takes into account not only the business perspective, but also the workload of an application and then assess its viability for Cloud migration. Enterprises need a tool that will enable them to draw an accurate assessment with real data to justify the categorization of the applications, along with a ROI/TCO comparison between an on-premise deployment and the Cloud.

Persistent has built a tool which we feel addresses the above mentioned challenges, click here for test drive and let us know your feedback.