Post Syndicated from Andy Klein original https://www.backblaze.com/blog/hard-drive-reliability-stats-q1-2016/

For Q1 2016 we are reporting on 61,590 operational hard drives used to store encrypted customer data in our data center. There are 9.5% more hard drives in this review versus our last review when we evaluated 56,224 drives. In Q1 2016, the hard drives in our data center, past and present, totaled over one billion hours in operation to date. That’s nearly 42 million days or 114,155 years worth of spinning hard drives. Let’s take a look at what these hard drives have been up to.

Backblaze hard drive reliability for Q1 2016

Below are the hard drive failure rates for Q1 2016. These are just for Q1 and are not cumulative, that chart is later.

Some observations on the chart:

1. The list totals 61,523 hard drives, not 61,590 noted above. We don’t list drive models in this chart of which we have less than 45 drives.
2. Several models have an annual failure rate of 0.00%. They had zero hard drive failures in Q1 2016.
3. Failure rates with a small number of failures can be misleading. For example, the 8.65% failure rate of the Toshiba 3TB drives is based on one failure. That’s not enough data to make a decision.
4. The overall Annual Failure Rate of 1.84% is the lowest quarterly number we’ve ever seen.

Cumulative hard drive reliability rates

We started collecting the data used in these hard drive reports on April 10, 2013, just about three years ago. The table below is cumulative as of 3/31 for each year since 4/10/2013.

One billion hours of spinning hard drives

Let’s take a look at what the hard drives we own have been doing for one billion hours. The one billion hours is a sum of all the data drives, past and present, in our data center. For example, it includes the WDC 1.0TB drives that were recently retired from service after an average of 6 years in operation. Below is a chart of hours in service to date ordered by drive hours:

The “Others” line accounts for the drives that are not listed because there are or were fewer than 45 drives in service.

In the table above, the Seagate 4TB drive leads in “hours in service” but which manufacturer has the most hours in service? The chart below sheds some light on this topic:

The early HGST drives, especially the 2- and 3TB drives, have lasted a long time and have provided excellent service over the past several years. This “time-in-service” currently outweighs the sheer quantity of Seagate 4 TB drives we have purchased and placed into service the last year or so.

Another way to look at drive hours is to see which drives, by size, have the most hours. You can see that in the chart below.

The 4TB drives have been spinning for over 580 million hours. There are 48,041 4TB drives which means each drive on average had 503 drive days of service, or 1.38 years. The annualized failure rate for all 4TB drives lifetime is 2.12%.

Hard Drive Reliability by Manufacturer

The drives in our data center come from four manufacturers. As noted above, most of them are from HGST and Seagate. With that in mind, here’s the hard drive failure rates by manufacturer, we’ve combined all of the drives, regardless of size, for a given manufacturer. The results are divided into one-year periods ending on 3/31 of 2014, 2015, and 2016.

Why are there less than 45 drives?

A couple of times we’ve noted that we don’t display drive models with fewer than 45 drives. Why would we have less than 45 drives given we need 45 drives to fill a Storage pod? Here are few of the reasons:

1. We once had 45 or more drives, but some failed and we couldn’t get replacements of that model and now we have less than 45.
2. They were sent to us as part of our Drive Farming efforts a few years back and we only got a few of a given model. We needed drives and while we liked using the same model, we utilized what we had.
3. We built a few Frankenpods that contained drives that were the same size in terabytes but had different models and manufacturers. We kept all the drives in a RAID array the same model, but there could be different models in each of the 3 RAID arrays in a given Frankenpod.

Regardless of the reason, if we have less than 45 drives of the same model, we don’t display them in the drive stats. We do however include their information in any “grand total” calculations such as drive space available, hours in service, failures, etc.

Buying drives from Toshiba and Western Digital

We often get asked why we don’t buy more WDC and Toshiba drives. The short answer is that we’ve tried. These days we need to purchase drives in reasonably large quantities, 5,000 to 10,000 at a time. We do this to keep the unit cost down and so we can reliably forecast our drive cost into the future. For Toshiba we have not been able to find their drives in sufficient quantities at a reasonable price. For WDC, we sometimes get offered a good price for the quantities we need, but before the deal gets done something goes sideways and the deal doesn’t happen. This has happened to us multiple times, as recently as last month. We would be happy to buy more drives from Toshiba and WDC, if we could, until then we’ll continue to buy our drives from Seagate and HGST.

What about using 6-, 8- and 10TB drives?

Another question that comes up is why the bulk of the drives we buy are 4TB versus the 5-, 6-, 8- and 10TB drives now on the market. The primary reason is that the price/TB for the larger drives is still too high, even when considering storage density. Another reason is availability of larger quantities of drives. To fill a Backblaze Vault built from 20 Storage Pod 6.0 servers, we need 1,200 hard drives. We are filling 3+Backblaze Vaults a month, but the larger size drives are hard to find in quantity. In short, 4TB drives are readily available at the right price, with 6- and 8TB drives getting close on price, but still limited in the quantities we need.

What is a failed hard drive?

For Backblaze there are three reasons a drive is considered to have “failed”:

1. The drive will not spin up or connect to the OS.
2. The drive will not sync, or stay synced, in a RAID Array (see note below).
3. The Smart Stats we use show values above our thresholds.

Note: Our stand-alone Storage Pods use RAID-6, our Backblaze Vaults use our own open-sourced implementation of Reed-Solomon erasure coding instead. Both techniques have a concept of a drive not syncing or staying synced with the other member drives in its group

A different look at Hard Drive Stats

We publish the hard drive stats data on our website with the Q1 2016 results there as well. Over the years thousands of people have downloaded the files. One of the folks who downloaded the data was Ross Lazarus, a self-described grumpy computational biologist. He analyzed the data using Kaplan-Meier statistics and plots, a technique typically used for survivability analysis. His charts and analysis present a different way to look at the data and we appreciate Mr. Lazarus taking the time to put this together. If you’ve done similar analysis of our data, please let us know in the comments section below – thanks.

The post One Billion Drive Hours and Counting: Q1 2016 Hard Drive Stats appeared first on Backblaze Blog | The Life of a Cloud Backup Company.

Post Syndicated from Ernesto original https://torrentfreak.com/megaupload-hard-drives-are-unreadable-hosting-company-warns-160518/

As we near the 5th anniversary of the Megaupload raid, data from hundreds of the site’s servers are gathering dust around the world.

This is also true for the files that were hosted by Cogent, one of the companies where Megaupload stored its servers.

While the original machines are no longer intact, the hosting company has backed up all data which it will keep in storage pending the various lawsuits against the company and its operators.

However, the lack of progress in the various legal proceedings isn’t doing the hard drives any good, something the RIAA and MPAA already hinted at earlier this year.

Not an unrealistic fear, as Cogent recently informed Megaupload and the rightsholders that half of the hard drives have now become unreadable.

“Recently, the parties have each been advised by Cogent that it has been unable to read eight of the sixteen computer hard drives on which the Megaupload cached data have been stored,” Megaupload informed (pdf) a Virginia federal court this week.

While this is a worrying message, it doesn’t necessarily mean that all data is lost. Cogent believes that the “drive heads” may be just be “frozen” but it has requested outside help to confirm this.

“Without the assistance of a computer forensic expert, however, Cogent cannot confirm that the data remains extant and uncorrupted,” Megaupload writes, adding that the hosting company doesn’t want to pay for the expenses itself.

On previous occasions the federal court postponed decisions over how to secure the evidence stored on hard drives, but all parties would now like to see some action.

In its request to the court, Megaupload argues that either the copyright holders or the Government should pick up the tab for preserving the data. The defunct file-hosting service can’t contribute itself, since its assets remain frozen.

Also, since the U.S. Government previously copied selected portions of the Cogent data as evidence, it now has an obligation to secure the rest as well, if only to avoid the suspicion of cherry-picking evidence.

“Having seized control of the Carpathia servers in order to obtain ‘selected’ portions of the data, the government has triggered its duty to preserve the remaining data because the entire data-set ‘might be significant’ to the defense of the Criminal Action,” Megaupload writes.

For their part the RIAA and MPAA also want to make sure that the data is preserved so have renewed their request (pdf) for a subpoena to obtain copies.

Alternatively, both rightsholder groups are open to bringing in an independent computer forensics vendor, to copy and preserve the data while the civil cases are on hold.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Ernesto original https://torrentfreak.com/torrzan-allows-download-torrents-via-telegram-160522/

With 100 million active users, Telegram is one of the most used messaging services, but it’s much more than that.

The application, which is supported by all leading operating systems, also allows developers to create nifty bots. This can come in quite handy.

The new TorrZan bot, for example, can grab torrents directly from the messaging application. With a series of simple commands it’s easy to find or add popular torrents, which can then be downloaded with a simple click.

The torrent files are not shared over the user’s local network, but from TorrZan’s servers. Once a download is complete the files can be downloaded to a local device, or played directly if they are streamable.

We tried the free version, which allows people to test three torrents up to 10GB without speed restrictions. Downloading a copy of the film Sintel took just a few minutes and after it was completed an https link to the mp4 video played seamlessly on both iPhone and PC.

The developers of TorrZan say they created the service to provide an easy and convenient tool for people to download torrents, while keeping user privacy in mind.

“We created this bot only because we believe that it is the most convenient and easiest way to download torrents across all platforms that are supported by Telegram, which includes Android, iOS, Windows Phone, PC, Mac and Linux,” TorrZan’s team informs TorrentFreak.

TorrZan doesn’t record or store any logs of your downloading activity, which is good to know. As an added privacy bonus, it uses external servers to download the torrents so users’ local IP-addresses are not visible to the public.

Besides adding torrents or magnet links manually, users can also use the bot to search for files. This appears to work nicely for well-known titles, and avoids the extra step of having to search for links on third-party sites.

“We integrated a search feature that displays torrents from popular torrent sites right inside our bot. This means that our users don’t have to jump through hoops, they can simply find torrents and download them directly inside the same interface,” the TorrZan team says.

The bot is still in the early stage of development and the team is planning to introduce several new features and updates in the future. Among other things, they want to make it easier to integrate video and audio playback with local devices.

For now, anyone can test the service for free without any restrictions. However, the services uses a freemium model and after three torrents non-paying users are limited to 1 GB of storage and a 50 KB\s download limit.

Those interested in giving it a free spin can visit the official TorrZan site.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Ernesto original https://torrentfreak.com/pirate-bay-24-hours-160524/

The Pirate Bay has been unreachable for more than a day now.

The Pirate Bay currently displays a CloudFlare error message across all domain names, confirming that TPB’s servers are unresponsive.

In addition, some proxy sites are also offline, as well as TPB’s .onion address which displays a blank page.

TorrentFreak reached out to the TPB team who are aware of the issues. They explained that problem is of a technical nature and hope to have the site back online soonish.

In any case, there’s no reason to panic.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-elastic-transcoder-update-support-for-mpeg-dash/

Amazon Elastic Transcoder converts media files (audio and video) from one format to another. The service is robust, scalable, cost-effective, and easy to use. You simply create a processing pipeline (pointing to a pair of S3 buckets for input and output in the process), and then create transcoding jobs. Each job reads a specific file from the input bucket, transcodes it to the desired format(s) as specified in the job, and then writes the output to the output bucket. You pay for only what you transcode, with price points for Standard Definition (SD) video, High Definition (HD) video, and audio. We launched the service with support for an initial set of transcoding presets (combinations of output formats and relevant settings). Over time, in response to customer demand and changes in encoding technologies, we have added additional presets and formats. For example, we added support for the VP9 Codec earlier this year.

Support for MPEG-DASH
Today we are adding support for transcoding to the MPEG-DASH format. This International Standard format supports high-quality audio and video streaming from HTTP servers, and has the ability to adapt to changes in available network throughput using a technique known as adaptive streaming. It was designed to work well across multiple platforms and at multiple bitrates, simplifying the transcoding process and sidestepping the need to create output in multiple formats.

During the MPEG-DASH transcoding process, the content is transcoded into segmented outputs at the different bitrates and a playlist is created that references these outputs. The client (most often a video player) downloads the playlist to initiate playback. Then it monitors the effective network bandwidth and latency, requests video segments as needed. If network conditions change during the playback process, the player will take action, upshifting or downshifting as needed.

You can serve up the transcoded content directly from S3 or you can use Amazon CloudFront to get the content even closer to your users. Either way, you need to create a CORS policy that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

If you are using CloudFront, you need to enable the OPTIONS method, and allow it to be cached:

You also need to add three headers to the whitelist for the distribution:

Transcoding With MPEG-DASH
To make use of the adaptive bitrate feature of MPEG-DASH, you create a single transcoding job and specify multiple outputs, each with a different preset. Here are your choices (4 for video and 1 for audio):

When you use this format, you also need to choose a suitable segment duration (in seconds). A shorter duration produces a larger number of smaller segments and allows the client to adapt to changes more quickly.

You can create a single playlist that contains all of the bitrates, or you can choose the bitrates that are most appropriate for your customers and your content. You can also create your own presets, using an existing one as a starting point:

Available Now
MPEG-DASH support is available now in all Regions where Amazon Elastic Transcoder is available. There is no extra charge for this use of this format (see Elastic Transcoder Pricing to learn more).

—
Jeff;

Post Syndicated from Andy original https://torrentfreak.com/fan-created-movie-subtitle-site-operator-facing-prison-160525/

Running a site offering or even linking to pirated movies and TV shows can be a hazardous occupation. It attracts the attention of copyright holders, the police, and in some cases even governments. For those running them these perils represent an occupational hazard.

But what if a site creates its own content and distributes that online, should that be a crime? That question is about to be answered in a unique case featuring fan-populated subtitling site Undertexter.se.

For ten years Undertexter (‘subtitles’ in Swedish) provided a somewhat useful service. Faced with what they perceived as a dearth of subtitling in local language, members of the site made their own translated subtitles for movies and TV shows. These were made available to all via the site.

However, in the summer of 2013 everything came crashing down. Under pressure from powerful Hollywood-based movie companies, police raided the site and seized its servers.

“The people who work on the site don’t consider their own interpretation of dialog to be something illegal, especially when we’re handing out these interpretations for free,” site founder Eugen Archy said at the time.

The authorities firmly disagreed, Archy was arrested, and the investigation into his site continued. Now, almost three years later, the Undertexter founder has been prosecuted for distributing infringing subtitles.

“I have indicted the person I say is behind the site Undertexter.se which made the dialogue from 74 films available to the public,” says prosecutor Henrik Rasmusson.

Of particular interest is the nature of the 74 movies referenced by the prosecution. Rather than tackle all of the subtitles on the site, the prosecution appears to have hand-picked a few dozen that gives them the strongest case, i.e those that relate to movies that weren’t commercially available in Sweden at the time.

The underlying suggestion is that those who created the subtitles either managed to legally view them in other regions or more likely carried out their translation work from pirate copies available online. Also, since the majority of Undertexter’s traffic came from Sweden, it’s likely that users of the site married the subtitles up with pirate copies.

Archy does not deny that he founded and operated the site, nor does he refute claims that he made some money from his activities, largely through on-site advertising. However, he does believe that offering fan-created subtitles is not a crime.

Unsurprisingly, Rasmusson strongly disagrees and even suggests that a prison sentence could be a possible outcome of this prosecution.

“This particular type of case, with pirate subtitles for pirate movies, has not been tried before. But the scale is at such a level that the penalty does not stop at fines, but imprisonment. It could be a suspended sentence,” Rasmusson says.

Soon it will be up to the court to decide whether distributing fan-created subtitles is a crime in Sweden. Experts have already weighed in on the case with Sanna Wolk, an associate professor of civil law at Uppsala University, noting that the devil could be in the detail.

“The core issue is whether the lyrics count as independent works or pure translations. If they follow the script, it’s a copyright violation to distribute them without permission, but if they’re self-published, it is not,” Wolk noted earlier.

“It is difficult to say where the exact line is. Subtitles need to be considered on their own merits to make an assessment.”

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Andy original https://torrentfreak.com/police-target-50-streaming-sites-detain-five-suspects-160530/

While torrents remain popular with millions of file-sharers, cheaper bandwidth and faster Internet connections have contributed to an explosion of content being streamed online.

Today there are thousands of sites offering huge libraries of unauthorized content, all of it available via a YouTube-like interface accessible via any Internet browser. With a non-existent learning curve, it’s piracy anyone can get involved in.

As a result these kinds of sites can quickly gain a massive following and efforts to hinder their operations continue every day. With millions of links being removed from search engines and site-blocking a regular occurrence, other more aggressive options are also regularly explored.

Currently that is the stance of prosecutors in Rome, Italy, who say they have carried out a large operation to shut down a network of sites offering live sports events, movies, TV shows and concerts without permission from copyright holders.

Titled Operation Match Off 2.0, the action was carried out by the Comando Unità Speciali (Special Command Unit) of the Guardia di Finanza (GdF), a department under Italy’s Minister of Economy and Finance tasked with dealing with financial crime.

According to GdF the operation targeted 50 sites running on 41 servers located on three continents. Three servers were seized locally in Italy. After raids were carried out in a number of regions across the country, five suspects were detained. Further details on the sites and the suspects are yet to be released.

The sites are said to have offered live streaming of sports, on-demand content such as movies and TV shows, plus scheduling features in return for a ten euro payment per month. Italian authorities say the equivalent official offering would cost nearer 100 euros.

GdF report that the five suspects had built of a “vast network” of users and were generating huge profits from them.

“To understand the scope of the operation we detected the presence of more than 340,000 registered users within a community,” GdF said in a statement.

“Assuming that everyone had an illegal ‘subscription’, you can, with a simple calculation, estimate that the turnover is nearly €3,500,000 monthly ($3.89m), or more than €40,000,000 ($44.46) per year.”

Should they be found guilty, the five suspects would face fines and up to four years in prison.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Ernesto original https://torrentfreak.com/10-years-ago-hollywood-awoke-the-pirate-bay-beast-160531/

Most of the site’s current users are probably unaware that without a few essential keystrokes in the site’s early years, The Pirate Bay may have not been here today.

May 31, 2006, less than three years after The Pirate Bay was founded, 65 Swedish police officers entered a datacenter in Stockholm.

The policemen had instructions to shut down the Pirate Bay’s servers, which were already seen as a major threat to the entertainment industry.

At the time The Pirate Bay wasn’t the giant it is today though. And ironically, the raid only made the site bigger, stronger and more resilient.

While the police were about to enter the datacenter, Pirate Bay founders Gottfrid and Fredrik got wind that something was up.

In the months before the raid they were already being watched by private investigators day and night, but this time something was about to happen to their trackers.

At around 10am in the morning Gottfrid told Fredrik that there were police officers at their office, and asked him to get down to the co-location facility and get rid of the ‘incriminating evidence,’ although none of it, whatever it was, was related to The Pirate Bay.

As Fredrik was leaving, he suddenly realized that the problems might be linked to their tracker. He therefore decided to make a full backup of the site, just in case.

When he later arrived at the co-location facility the concerns turned out to be justified. There were dozens of policemen floating around taking away dozens of servers, most of which belonged to clients unrelated to The Pirate Bay.

In the days that followed it became clear that Fredrik’s decision to start a backup of the site was probably the most pivotal moment in the site’s history. Because of this backup Fredrik and the rest of the Pirate Bay team managed to resurrect the site within three days.

Of course, the entire situation was handled with the mockery TPB had become known for.

Unimpressed, the site’s operators renamed the site “The Police Bay” complete with a new logo shooting cannon balls at Hollywood. A few days later this logo was replaced by a Phoenix, a reference to the site rising from its digital ashes.

Instead of shutting it down the raid brought the site into the mainstream press, not least due to its swift resurrection. All the publicity also triggered a huge traffic spike for TPB, exactly the opposite effect Hollywood had hoped for.

Despite a criminal investigation leading to convictions for the site’s founders, The Pirate Bay kept growing and growing in the years that followed.

The site’s assets, meanwhile, were reportedly transferred to the Seychelles-based company Reservella.

Under new ownership several major technical changes occurred. In the fall of 2009 the infamous BitTorrent tracker was taken offline, turning The Pirate Bay into a torrent indexing site.

Early 2012 The Pirate Bay went even further when it decided to cease offering torrent files for well-seeded content. The site’s operators moved to magnet links instead, allowing them to save resources while making it easier for third-party sites to run proxies.

These proxies turned out to be much-needed, as The Pirate Bay is now the most broadly censored website on the Internet. In recent years ISPs all around the world have been ordered by courts to block subscriber access to the torrent site.

While TPB swiftly recovered from the “original” raid, it did suffer nearly two months of downtime late 2014 when another raid took place.

Initially it was believed that some of the site’s crucial servers were taken by the police, but the TPB team later said that it was barely hit and that they took the site offline as a precaution.

While the first raid make The Pirate Bay stronger, the two-month stint of downtime was a big hit. While the site still has millions of visitors per day, it is no longer the most dominant player, and is still suffering from regular outages.

That said, The Pirate Bay is expected to live on and on. To celebrate its turbulent past the site’s operators declared May 31 to be Pirate Independence Day a few years ago.

“Let today be the pirates’ Independence Day! Today we celebrate the victories we’ve had and the victories that will come. Today we celebrate that we’re united in our efforts. Keep on seeding!” the TPB team said at the time.

But remember, if there hadn’t been a recent backup back in 2006, things may have turned out quite differently.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/05/from-scratch-why-these-mass-scans-are.html

The way the Internet works is that “packets” are sent to an “address”. It’s the same principle how we send envelopes through the mail. Just put an address on it, hand it to the nearest “router”, and the packet will get forwarded hop-to-hop through the Internet in the direction of the destination.

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-in-aws-marketplace-alces-flight-effortless-hpc-on-demand/

In the past couple of years, academic and corporate researchers have begun to see the value of the cloud. Faced with a need to run demanding jobs and to deliver meaningful results as quickly as possible while keeping costs under control, they are now using AWS to run a wide variety of compute-intensive, highly parallel workloads.

Instead of fighting for time on a cluster that must be shared with other researchers, they accelerate their work by launching clusters on demand, running their jobs, and then shutting the cluster down shortly thereafter, paying only for the resources that they consume. They replace tedious RFPs, procurement, hardware builds and acceptance testing with cloud resources that they can launch in minutes. As their needs grow, they can scale the existing cluster or launch a new one.

This self-serve, cloud-based approach favors science over servers and accelerates the pace of research and innovation. Access to shared, cloud-based resources can be granted to colleagues located on the same campus or halfway around the world, without having to worry about potential issues at organizational or network boundaries.

Alces Flight in AWS Marketplace
Today we are making Alces Flight available in AWS Marketplace. This is a fully-featured HPC environment that you can launch in a matter of minutes. It can make use of On-Demand or Spot Instances and comes complete with a job scheduler and hundreds of HPC applications that are all set up and ready to run. Some of the applications include built-in collaborative features such as shared graphical views. For example, here’s the Integrative Genomics Viewer (IGV):

Each cluster is launched into a Virtual Private Cloud (VPC) with SSH and graphical desktop connectivity. Clusters can be of fixed size, or can be Auto Scaled in order to meet changes in demand.  Once launched, the cluster looks and behaves just like a traditional Linux-powered HPC cluster, with shared NFS storage and passwordless SSH access to the compute nodes. It includes access to HPC applications, libraries, tools, and MPI suites.

We are launching Alces Flight in AWS Marketplace today. You can launch a small cluster (up to 8 nodes) for evaluation and testing or a larger cluster for research.

If you subscribe to the product, you can download the AWS CloudFormation template from the Alces site. This template powers all of the products, and is used to quickly launch all of the AWS resources needed to create the cluster.

EC2 Spot Instances give you access to spare AWS capacity at up to a 90% discount from On-Demand pricing and can significantly reduce your cost per core. You simply enter the maximum bid price that you are willing to pay for a single compute node; AWS will manage your bid, running the nodes when capacity is available at the desired price point.

Running Alces Flight
In order to get some first-hand experience with Alces Flight, I launched a cluster of my own. Here are the settings that I used:

I set a tag for all of the resources in the stack as follows:

I confirmed my choices and gave CloudFormation the go-ahead to create my cluster. As expected, the cluster was all set up and ready to go within 5 minutes. Here are some of the events that were logged along the way:

Then I SSH’ed in to the login node and saw the greeting, all as expected:

After I launched my cluster I realized that this post would be more interesting if I had more compute nodes in my cluster. Instead of starting over, I simply modified my CloudFormation stack to have 4 nodes instead of 1, applied the change, and watched as the new nodes came online. Since I specified the use of Spot Instances when I launched the cluster, Auto Scaling placed bids automatically. Once the nodes were online I was able to locate them from within my PuTTY session:

Then I used the pdsh (Parallel Distributed Shell command) to check on the up-time of each compute node:

Learn More
This barely counts as scratching the surface; read Getting Started as Quickly as Possible to learn a lot more about what you can do! You should also watch one or more of the Alces videos to see this cool new product in action.

If you are building and running data-intensive HPC applications on AWS, you may also be interested in another Marketplace offering. The BeeGFS (self-supported or support included) parallel file system runs across multiple EC2 instances, aggregating the processing  power into a single namespace, with all data stored on EBS volumes.  The self-supported product is also available on a 14 day free trial. You can create a cluster file system using BeeGFS and then use it as part of your Alces cluster.

—
Jeff;

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2016/06/stealth_falcon_.html

Citizen Lab has the details:

This report describes a campaign of targeted spyware attacks carried out by a sophisticated operator, which we call Stealth Falcon. The attacks have been conducted from 2012 until the present, against Emirati journalists, activists, and dissidents. We discovered this campaign when an individual purporting to be from an apparently fictitious organization called “The Right to Fight” contacted Rori Donaghy. Donaghy, a UK-based journalist and founder of the Emirates Center for Human Rights, received a spyware-laden email in November 2015, purporting to offer him a position on a human rights panel. Donaghy has written critically of the United Arab Emirates (UAE) government in the past, and had recently published a series of articles based on leaked emails involving members of the UAE government.

Circumstantial evidence suggests a link between Stealth Falcon and the UAE government. We traced digital artifacts used in this campaign to links sent from an activist’s Twitter account in December 2012, a period when it appears to have been under government control. We also identified other bait content employed by this threat actor. We found 31 public tweets sent by Stealth Falcon, 30 of which were directly targeted at one of 27 victims. Of the 27 targets, 24 were obviously linked to the UAE, based on their profile information (e.g., photos, “UAE” in account name, location), and at least six targets appeared to be operated by people who were arrested, sought for arrest, or convicted in absentia by the UAE government, in relation to their Twitter activity.

The attack on Donaghy — and the Twitter attacks — involved a malicious URL shortening site. When a user clicks on a URL shortened by Stealth Falcon operators, the site profiles the software on a user’s computer, perhaps for future exploitation, before redirecting the user to a benign website containing bait content. We queried the URL shortener with every possible short URL, and identified 402 instances of bait content which we believe were sent by Stealth Falcon, 73% of which obviously referenced UAE issues. Of these URLs, only the one sent to Donaghy definitively contained spyware. However, we were able to trace the spyware Donaghy received to a network of 67 active command and control (C2) servers, suggesting broader use of the spyware, perhaps by the same or other operators.

News story.

Post Syndicated from Jason DeWeese original https://blogs.aws.amazon.com/security/post/Tx1AO9IK88HEIBI/How-to-Set-Up-DNS-Resolution-Between-On-Premises-Networks-and-AWS-by-Using-Unbou

In previous AWS Security Blog posts, Drew Dennis covered two options for establishing DNS connectivity between your on-premises networks and your Amazon Virtual Private Cloud (Amazon VPC) environments. His first post explained how to use Simple AD to forward DNS requests originating from on-premises networks to an Amazon Route 53 private hosted zone. His second post showed how you can use Microsoft Active Directory (also provisioned with AWS Directory Service) to provide the same DNS resolution with some additional forwarding capabilities.

In this post, I will explain how you can set up DNS resolution between your on-premises DNS with Amazon VPC by using Unbound, an open-source, recursive DNS resolver. This solution is not a managed solution like Microsoft AD and Simple AD, but it does provide the ability to route DNS requests between on-premises environments and an Amazon VPC–provided DNS.

Overview of Unbound

Unbound allows resolution of requests originating from AWS by forwarding them to your on-premises environment—and vice versa. For the purposes of this post, I will focus on a basic installation of Amazon Linux with the configuration necessary to direct traffic to on-premises environments or to the Amazon VPC–provided DNS, as appropriate. Review the Unbound documentation for details and other configuration options.

The following diagrams show an AWS architecture that uses Unbound to forward DNS traffic. The first diagram illustrates requests originating from AWS. Traffic matching the on-premises domain is redirected to the on-premises DNS server. All traffic not matching the on-premises domain will be forwarded to the Amazon VPC–provided DNS.

The second diagram illustrates requests originating from an on-premises environment. The on-premises environment forwards traffic to Unbound, which in turn forwards the traffic to the Amazon VPC–provided DNS.

Step 1: Install Unbound on Amazon EC2

To make the installation of Unbound as automated as possible, you will use EC2 user data to run shell commands at launch. The easiest way to do this is by creating a new EC2 instance.

To create an EC2 instance with Unbound:

1. In the AWS Management Console, click EC2 under Compute.
2. From the EC2 console, click Launch Instance.
3. In Step 1: Choose an Amazon Machine Image (AMI), select the latest 64-bit Amazon Linux Amazon Machine Image (AMI), which should be at the top of the Quick Start list.
4. In Step 2: Choose an Instance Type, select an instance with enough compute capacity to handle your DNS traffic. An m4.large or m3.medium instance type would be a good place to start.
5. On the Step 3: Configure Instance Details page, (see the following screenshot) select an Amazon VPC and Subnet. If you have a preferred private Primary IP for the instance, type it under the Network Interface section. If you do not set a private Primary IP, Amazon VPC will assign one. Take note of the IP address of the instance because you will need it in a later step.

6. While still on the Configuration Instance Details page, paste the following shell script in the User data box As text, as show in the preceding screenshot. Update the vpc_dns, onprem_domain, and onprem_dns variables in the script to reflect your DNS servers on-premises and in the Amazon VPC as well as the domain name you use for on-premises. Note that the Amazon VPC–provided DNS IP address will always be your Amazon VPC CIDR block “plus two.” For example, if your Amazon VPC uses 198.51.100.0/24, the VPC-provided DNS is 198.51.100.2.

#!/bin/bash
# Set the variables for your environment
vpc_dns=198.51.100.2
onprem_domain=example.local
onprem_dns=192.0.2.2

# Install updates and dependencies
yum update -y
yum install -y gcc openssl-devel expat-devel
# Get, build, and install latest Unbound
wget https://unbound.net/downloads/unbound-latest.tar.gz
tar -zxvf unbound-latest.tar.gz
cd unbound-*
./configure && make && make install
# Add run-time user
useradd unbound

# Write Unbound configuration file with values from variables
cat << EOF | tee /usr/local/etc/unbound/unbound.conf
server:
        interface: 0.0.0.0
        access-control: 0.0.0.0/0 allow
forward-zone:
        name: "."
        forward-addr: ${vpc_dns}
forward-zone:
        name: "${onprem_domain}"
        forward-addr: ${onprem_dns}
EOF

# Install Unbound as service and run
cat << EOF | tee /etc/init/unbound.conf
start on runlevel [2345]
exec /usr/local/sbin/unbound
EOF

start unbound

When the script runs, it installs Unbound with all its dependencies, creates a configuration file using the values you have supplied, and configures the Unbound service to launch on subsequent instance reboots.

7. In Step 4: Add Storage, leave the defaults selected.
8. In Step 5: Tag Instance, type Unbound DNS as the Value for the Name.
9. In Step 6: Configure Security Group, select Create a new security group (as shown in the following screenshot) and type unbound-dns in the Security group name box. Select DNS (UDP) as the Type, and select Custom IP for Source. Enter the Amazon VPC CIDR as the Custom IP (for example, 198.51.100.0/24). The VPC CIDR is available in the Network list in Step 3: Configure Instance of the Launch Instance wizard. This will allow DNS traffic to flow to the server.

Repeat these steps to install Unbound on at least two EC2 instances in different Availability Zones in order to provide redundant DNS servers.

Step 2: Configure your EC2 instances to use Unbound

Now that you have an instance of Unbound running in Amazon VPC, you now have to configure the EC2 instance to use Unbound as the DNS server so that on-premises domain names can be resolved. DHCP options sets allow you to assign the domain name, domain name servers, and other DHCP options.

1. In the AWS Management Console, click VPC under Networking.
2. Click DHCP Options Sets in the left pane and then click Create DHCP options set.
3. In Name tag box, type Unbound DNS.
4. In Domain name servers box, type the IP addresses of the Unbound instances you noted in Step 5 when creating the Unbound instance. Separate multiple entries with commas.
5. Click the Yes, Create button to create the DHCP options set.

6. Associate the DHCP options set with your Amazon VPC by clicking Your VPCs in the left pane of the VPC console.
7. Right-click the Amazon VPC with which you want to use Unbound, and then select the DHCP options set you just created.

Multiple Amazon VPCs in a single region can use an Unbound DNS server across an Amazon VPC peering connection, which allows Amazon VPC to host Unbound as a shared service with other Amazon VPCs. You must make sure that the proper routing rules are created and the security group assigned to the Unbound instance is configured to allow traffic inbound from the peered Amazon VPCs. For more information, see Peering to One VPC to Access Centralized Resources.

Step 3: Configure on-premises DNS to forward to Unbound

For on-premises resources to resolve domain names assigned to AWS resources, you must take additional steps to configure your on-premises DNS server to forward requests to Unbound. Ensure the following are configured:

• Your on-premises DNS has a forwarder that directs requests for the AWS-hosted domains to EC2 instances running Unbound . Refer to the documentation for your on-premises DNS server to configure DNS forwarders.
• You have an Amazon VPN or AWS Direct Connect with routing rules that allow DNS traffic to pass through to the Amazon VPC.
• The security group assigned to Unbound instances allows traffic from your on-premises DNS server that will forward requests. Revisit #9 in Step 1: Install Unbound on Amazon EC2. Add an entry with the on-premises CIDR that allows DNS (UDP).

Summary

You can use Unbound as a DNS forwarder to create an architecture such that DNS requests originating from your on-premises environment or your Amazon VPCs can be resolved. After you have correctly configured the setup detailed in this post, it will provide integration between DNS services.

If you have comments, submit them in the “Comments” section below. If you have questions, start a new thread on the Directory Service forum.

– Jason

Post Syndicated from Jonathan Fritz original https://blogs.aws.amazon.com/bigdata/post/Tx2ZF1NDQYDJFGT/Supercharge-SQL-on-Your-Data-in-Apache-HBase-with-Apache-Phoenix

With today’s launch of Amazon EMR release 4.7, you can now create clusters with Apache Phoenix 4.7.0 for low-latency SQL and OLTP workloads. Phoenix uses Apache HBase as its backing store (HBase 1.2.1 is included on Amazon EMR release 4.7.0), using HBase scan operations and coprocessors for fast performance. Additionally, you can map Phoenix tables and views to existing HBase tables, giving you SQL access over data already stored in HBase.

Let’s run through a quick demo to explore how to connect to Phoenix using JDBC, create a view over an existing HBase table, and create a secondary index for increased read performance.

Create an Amazon EMR cluster and an HBase table

First, using the Amazon EMR console or AWS CLI, launch a new Amazon EMR cluster using release 4.7 and choose Phoenix as an application. Here’s an example AWS CLI command:

aws emr create-cluster --name PhoenixDemo --release-label emr-4.7.0 --instance-type m3.xlarge --instance-count 3 --applications Name=Phoenix --ec2-attributes KeyName=MyKeyName --use-default-roles

Selecting the Phoenix application also includes HBase and Hadoop (YARN, HDFS, and MapReduce), giving you all the components needed for a fully operational cluster.

Next, create a table in HBase to use with Phoenix. You will copy an HBase snapshot from Amazon S3 and restore it on your cluster. Go to this HBase post on the AWS Big Data Blog and follow the instructions under the  “HBase shell query walkthrough” section to restore a table named customer (3,448,682 rows).

Finally, run a get request example from that blog to certify your table has been restored correctly.

Connect to Phoenix using JDBC and create a table

Once your HBase table is ready, it’s time to map a table in Phoenix to your data in HBase. You use a JDBC connection to access Phoenix, and there are two drivers included on your cluster under /usr/lib/phoenix/bin. First, the Phoenix client connects directly to HBase processes to execute queries, which requires several ports to be open in your Amazon EC2 Security Group (for ZooKeeper, HBase Master, and RegionServers on your cluster) if your client is off-cluster.

Second, the Phoenix thin client connects to the Phoenix Query Server, which runs on port 8765 on the master node of your EMR cluster. This allows you to use a local client without adjusting your Amazon EC2 Security Groups by creating a SSH tunnel to the master node and using port forwarding for port 8765. The Phoenix Query Server is still a new component, and not all SQL clients can support the Phoenix thin client.

In this example, you will use the SQLLine client included with Phoenix on the master node to connect to the Phoenix Query Server. Return to the terminal on the master node of your cluster. If you closed your SSH tunnel after creating your HBase table, create another SSH tunnel. Connect to Phoenix using this command:

/usr/lib/phoenix/bin/sqlline-thin.py http://localhost:8765

Once the SQLLine client has connected, let’s create a SQL view over the customer table in HBase. We will create a view instead of a table, because dropping a view does not also delete the underlying data in HBase (the behavior for deleting underlying data in HBase for Phoenix tables is configurable, but is true by default). To map a pre-existing table in HBase, you use a ‘column_family’.’column_prefix’ format for each column you want to include in your Phoenix view (note that you must use quotation marks around column and table names that are lowercase). Also, identify the column that is the HBase primary key with PRIMARY KEY, and give the view the same name as the underlying HBase table. Now, create a view over the customer table:

CREATE VIEW "customer" (
pk VARCHAR PRIMARY KEY,
"address"."state" VARCHAR,
"address"."street" VARCHAR,
"address"."city" VARCHAR,
"address"."zip" VARCHAR,
"cc"."number" VARCHAR,
"cc"."expire" VARCHAR,
"cc"."type" VARCHAR,
"contact"."phone" VARCHAR);

Use SQLLine’s !tables command to list available Phoenix tables and confirm your newly created view is in the list . Make sure your terminal window is wide enough to show the output before instantiating the SQLLine client. Otherwise, the complete output will not appear.

Speeding up queries with secondary indexes

First, run a SQL query counting the number of people with each credit card type in California:

SELECT "customer"."type" AS credit_card_type, count(*) AS num_customers FROM "customer" WHERE "customer"."state" = 'CA' GROUP BY "customer"."type";

However, because we aren’t including the Primary Key in the HBase table in the WHERE clause, Phoenix must scan all HBase rows to ensure that all rows with the state ‘CA’ are included. If we anticipate our read patterns will filter by state, we can create a secondary index on that column to give Phoenix the ability to scan along that axis. For a more in-depth view of secondary indexing feature set, see the Apache Phoenix documentation. Now create a covered secondary index on state and include the HBase primary key (the customer ID), city, expire date, and type:

CREATE INDEX my_index ON "customer" ("customer"."state") INCLUDE("PK", "customer"."city", "customer"."expire", "customer"."type");

Phoenix will use a Hadoop MapReduce job to create this index and parallelly load it into HBase as another table (this takes around 2 minutes). Now, rerun the SQL query from earlier and compare the performance. It should be at least 10x faster!

Conclusion

In this post, you learned how to connect to Phoenix using JDBC, create Phoenix views over data in HBase, create secondary indexes for faster performance, and query data. You can use Phoenix as a performant SQL interface over existing HBase tables or use Phoenix directly to populate and manage tables using HBase behind the scenes as an underlying data store. To learn more about Phoenix, see the Amazon EMR documentation or the Apache documentation.

If you have any questions about using Phoenix on Amazon EMR or would like to share interesting use cases that leverage Phoenix, please leave a comment below.

——————————-

Related

Combine NoSQL and Massively Parallel Analytics Using Apache HBase and Apache Hive on Amazon EMR

Want to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.

Post Syndicated from n8willis original http://lwn.net/Articles/689792/rss

At his blog, Gunnar Wolf urges developers to stop using
“short” (eight hex-digit) PGP key IDs as soon as possible. The
impetus for the advice originates with Debian’s Enrico Zini, who recently
found two keys sharing the same short ID in the wild. The
possibility of short-ID collisions has been known for a while, but it
is still disconcerting to see in the wild. “Those three keys
are not (yet?) uploaded to the keyservers, though… But we can expect
them to appear at any point in the future. We don’t know who is behind
this, or what his purpose is. We just know this looks very
evil.”

Wolf goes on to note that short IDs are not merely human-readable
conveniences, but are actually used to identify PGP keys in some
software programs. To mitigate the risk, he recommends configuring
GnuPG to never shows short IDs, to ensure that other programs do not
consume short IDs, and to “only sign somebody else’s key if you
see and verify its full fingerprint. […] And there are surely many other important recommendations. But this is a good set of points to start with.”

Post Syndicated from jake original http://lwn.net/Articles/690691/rss

The first version of KDE neon, which is a distribution based on Ubuntu 16.04 that is meant to be a stable platform on which to try the latest Plasma desktop, has been released.
“KDE neon User Edition 5.6 is based on the latest version of Plasma 5.6 and intends to showcase the latest KDE technology on a stable foundation. It is a continuously updated installable image that can be used not just for exploration and testing but as the main operating system for people enthusiastic about the latest desktop software. It comes with a slim selection of apps, assuming the user’s capacity to install her own applications after installation, to avoid cruft and meaningless weight to the ISO. The KDE neon team will now start adding all of KDE’s applications to the neon archive.

Since the announcement of the project four months ago the team has been working on rolling out our infrastructure, using current best-practice devops technologies. A continuous integration Jenkins system scans the download servers for new releases and automatically fires up computers with Docker instances to build packages. We work in the open and as a KDE project any KDE developer has access to our packaging Git repository and can make fixes, improvements and inspect our work.”

Post Syndicated from Ernesto original https://torrentfreak.com/judge-failing-megaupload-servers-should-be-repaired-not-copied-160611/

Megaupload was shutdown nearly half a decade ago, but data from hundreds of the site’s servers are still in storage.

This is also true for the files that were placed at Cogent.

While the original machines are no longer intact, the hosting company has backed up all data which it will keep in storage pending the various lawsuits against the company and its operators.

However, as time has gone by the condition of the harddrives has deteriorated. A few weeks ago Cogent warned that sixteen of them have actually become unreadable, which is a serious concern since they contain important evidence.

To resolve the issue the RIAA and MPAA, representing various major copyright holders, asked if they could preserve a copy of the data themselves. Alternatively, they were also open to bringing in an independent computer forensics vendor, to copy and preserve the data.

Megaupload disagreed, arguing that rightsholders or other outsiders shouldn’t get their hands on possibly privacy sensitive user data, and opted to simply repair the failing disks.

This week District Court Judge Liam O’Grady ruled on the matter at hand. He rejected the copying proposal by the rightsholders, and went with Megaupload’s suggestion instead.

“The Court finds Defendants’ proposal is the more appropriate remedy for the issue at hand,” Judge O’Grady writes in his order.

The Judge instructs all stakeholders in the civil and criminal cases, including the U.S. Government and Cogent, to come together and agree on a repair process.

“[All parties] shall meet and confer with United States Magistrate Judge John F. Anderson to discuss and devise an appropriate solution to repair the Cogent drives and preserve the evidence on the Cogent servers, as well as to secure and preserve other digital evidence.”

While none of the parties are likely to disagree to a repair, they do have to determine who should pick up the tab.

Megaupload previously said that it doesn’t have the financial resources to do so, and suggested that either the copyright holders or the Government must take care of this. The Government is unlikely to pay though, and previously said that it no longer has an interest in the data.

The fact that the recent filings in the Megaupload proceedings are about data loss is exemplary of the slow progress in the cases, which are still a long way from trial.

Last December a New Zealand District Court judge ruled that Kim Dotcom and his colleagues can be extradited to the United States to face criminal charges. This decision was appealed and will be heard later this summer, so until then not much is expected to happen.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/06/scanning-for-clamav-0day.html

Last week an 0day was released for ClamAV. Well, not really an 0day so much as somebody noticed idiotic features in ClamAV. So I scanned the Internet for the problem.

The feature is that the daemon listens for commands that tell it to do things like scan files. Normally, it listens only locally for such commands, but can be reconfigured to listen remotely on TCP port 3310. Some packages that include ClamAV sometimes default to this.

It’s a simple protocol that consists of sending a command in clear text, like “PING”, “VERSION”, “SHUTDOWN”, or “SCAN
So I ran masscan with the following command:

masscan 0.0.0.0/0 -p3310 –banners –hello-string[3310] VkVSU0lPTg==

Normally when you scan an address range (/0) and port (3310), you’d just see which ports are open/closed. That’s not useful in this case, because it finds 2.7 million machines. Instead, you want to establish a full TCP connection. That’s what the –banners option does, giving us only 38 thousand machines that successfully establish a connection. The remaining machines are large ranges on the Internet where firewalls are configured to respond with SYN-ACK, with the express purpose of frustrating port scanners.

But of those 38k machines, most are actually things like web servers running on odd ports. 51 machines running VNC, 641 machines running SSH, and so on.

To find specifically ClamAV, I send a command using the –hello-string feature. I send the text “VERSION“, which must be encoded with base64 on the command-line for masscan (in case you need to also send binary).

This finds 5950 machines (i.e. 6k) that respond back with a ClamAV signature. typical examples of this response are:

At first I thought the date was when they last updated the software, maybe as a page. Roughly half had dates of either this morning or the day before. But no, it’s actually the dates when they last updated their signatures.

From this we can conclude that roughly half of ClamAV installations are configured to auto-update their signatures.

Roughly 2400 machines (nearly half) had the version 0.97.5. This was released in June 2012 (four years old). I’m thinking some appliance maker like Barracuda bundled the software — appliances are notorious for not getting updated software. That hints at why this non-default configuration is so common — it’s not users who made this decision, but the software that bundles ClamAV with other things. Scanning other ports gives me no clues — they appear all over the map, with different versions of SSH, different services running, different SSL versions, and so on. I thought maybe “mail server” (since that’d be a common task for ClamAV), but there were only a few servers, and they ran different mail server software. So it’s a mystery why this specific version is so popular.

I manually tested various machines with “SCAN foo”. They all replied “file not found”, which hints that all the units I found are vulnerable to this 0day.

As for other things, I came across a bunch of systems claiming to be ChinaDDoS systems:

Conclusion

This sort of stuff shouldn’t exist. The number of ClamAV systems available on the public Internet should be zero.

Even inside a corporate network, the number should be 0. If that stuff is turned on, then it should be firewalled (such as with iptables) so that only specific machines can access it.

Two important results are that half the systems are really old (EOLed, no longer supported), and only half the systems have the latest updates. There’s some overlap — systems with latest signature but out-of-date software.

Post Syndicated from Ernesto original https://torrentfreak.com/u-s-isp-sues-music-group-over-piracy-allegations-160615/

With 400,000 subscribers nationwide, RCN is one of the larger Internet providers in the United States.

Like many other ISPs the company has been overloaded with piracy notices in recent years. One of the most prolific senders is Rightscorp, who submit DMCA notices on behalf of clients including BMG.

These notices are controversial, because they use an aggressive tone paired with settlement demands.

In addition, Rightscorp and its clients claim that ISPs could be held liable for the infringing actions of their customers if they fail to take proper action. This includes disconnecting repeat copyright infringers.

RCN is not pleased with these allegations and this week took legal action. The Internet provider filed a lawsuit against music rights group BMG at a New York federal court, seeking a legal opinion on the matter.

“The central question for this Court’s determination is whether an Internet service provider should be held liable for copyright infringement simply because it provides Internet connectivity to its customers,” RCN writes.

The Internet provider explains that BMG and its anti-piracy partner are demanding payment for the alleged wrongdoings of its customers. In the process, they are bombarding RCN’s mailservers with notices.

“Both BMG and Rightscorp are wrongly demanding payment from RCN for that alleged infringement, and have clearly expressed their intention to enforce these purported rights,” the ISP writes.

“To substantiate its allegations, BMG asserts that RCN is on notice of the alleged wrongdoing by pointing to Rightscorp’s history of inundating RCN’s email server with millions of notifications purportedly reflecting instances of subscriber infringement.”

According to the Internet provider the notices are so numerous and so lacking in specificity, that it’s not feasible to investigate the claims. In addition, RCN points out that Rightscorp’s monitoring technology is flawed for various reasons.

Among other things, the ISP notes that Rightscorp only checks if a small portion of an alleged copyrighted work is shared, not the entire file.

RCN further says that it is not liable for the infringement of its subscribers because it is merely passing on traffic, which allows the company protection under the DMCA’s safe harbor provision.

The company is asking the court to review the matter and issue a declaratory judgment to provide more certainty.

“BMG’s repeated assertions that RCN is liable for copyright infringement lack merit. RCN therefore seeks a judgment from this Court declaring that it is not liable to BMG for copyright infringement,” RCN writes.

This is not the first lawsuit to deal with the question of liability.

In a similar case last year, Internet provider Cox Communications was held responsible for the copyright infringements of its subscribers. In that case a Virginia federal court ordered Cox to pay BMG $25 million in damages.

Given the stakes at hand, it wouldn’t be a surprise to see various other ISPs and copyright holders taking an interest in RCN’s case, as it’s likely to have a wide impact.

The full complaint filed by RCN is available here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Post Syndicated from BrianKrebs original https://krebsonsecurity.com/2016/06/fbi-raids-spammer-outed-by-krebsonsecurity/

Michael A. Persaud, a California man profiled in a Nov. 2014 KrebsOnSecurity story about a junk email artist currently flagged by anti-spam activists as one of the world’s Top 10 Worst Spammers, was reportedly raided by the FBI in connection with a federal spam investigation.

According to a June 9 story at ABC News, on April 27, 2016 the FBI raided the San Diego home of Persaud, who reportedly has been under federal investigation since at least 2013. The story noted that on June 6, 2016, the FBI asked for and was granted a warrant to search Persaud’s iCloud account, which investigators believe contained “evidence of illegal spamming’ and wire fraud to further [Persaud’s] spamming activities.”

Persaud doesn’t appear to have been charged with a crime in connection with this investigation. He maintains his email marketing business is legitimate and complies with the CAN-SPAM Act, the main anti-spam law in the United States which prohibits the sending of spam that spoofs that sender’s address or does not give recipients an easy way to opt out of receiving future such emails from that sender.

The affidavit that investigators with the FBI used to get a warrant for Persaud’s iCloud account is sealed, but a copy of it was obtained by KrebsOnSecurity. It shows that during the April 2016 FBI search of his home, Persaud told agents that he currently conducts internet marketing from his residence by sending a million emails in under 15 minutes from various domains and Internet addresses.

The affidavit indicates the FBI was very interested in the email address michaelp77x@gmail.com. In my 2014 piece Still Spamming After All These Years, I called attention to this address as the one tied to Persaud’s Facebook account — and to 5,000 or so domains he was advertising in spam. The story was about how the junk email Persaud acknowledged sending was being relayed through broad swaths of Internet address space that had been hijacked from hosting firms and other companies.

FBI Special Agent Timothy J. Wilkins wrote that investigators also subpoenaed and got access to that michaelp77x@gmail.com account, and found emails between Persaud and at least four affiliate programs that hire spammers to send junk email campaigns.

A spam affiliate program is a type of business or online retailer — such as an Internet pharmacy — that pays a third party (known as affiliates or spammers) a percentage of any sales that they generate for the program (for a much deeper dive on how affiliate programs work, check out Spam Nation).

When I wrote about Persaud back in 2014, I noted that his spam generally advertised the types of businesses you might expect to see pimped in junk email: payday loans, debt consolidation services, and various “nutraceutical” products.

Persaud did not respond to requests for comment. But in an email he sent to KrebsOnSecurity in November 2014, he said:

“I can tell you that my company deals with many different ISPs both in the US and overseas and I have seen a few instances where smaller ones will sell space that ends up being hijacked,” Persaud wrote in an email exchange with KrebsOnSecurity. “When purchasing IP space you assume it’s the ISP’s to sell and don’t really think that they are doing anything illegal to obtain it. If we find out IP space has been hijacked we will refuse to use it and demand a refund. As for this email address being listed with domain registrations, it is done so with accordance with the CAN-SPAM guidelines so that recipients may contact us to opt-out of any advertisements they receive.”

Persaud is currently listed as #10 on the World’s 10 Worst Spammers list maintained by Spamhaus, an anti-spam organization. In 1998, Persaud was sued by AOL, which charged that he committed fraud by using various names to send millions of get-rich-quick spam messages to America Online customers. In 2001, the San Diego District Attorney’s office filed criminal charges against Persaud, alleging that he and an accomplice crashed a company’s email server after routing their spam through the company’s servers.

Post Syndicated from Matthew Garrett original http://mjg59.dreamwidth.org/43486.html

I bought some awful WiFi lightbulbs a few months ago. The short version: they introduced terrible vulnerabilities on your network, they violated the GPL and they were also just bad at being lightbulbs. Since then I’ve bought some other Internet of Things devices, and since people seem to have a bizarre level of fascination with figuring out just what kind of fractal of poor design choices these things frequently embody, I thought I’d oblige.

Today we’re going to be talking about the KanKun SP3, a plug that’s been around for a while. The idea here is pretty simple – there’s lots of devices that you’d like to be able to turn on and off in a programmatic way, and rather than rewiring them the simplest thing to do is just to insert a control device in between the wall and the device andn ow you can turn your foot bath on and off from your phone. Most vendors go further and also allow you to program timers and even provide some sort of remote tunneling protocol so you can turn off your lights from the comfort of somebody else’s home.

The KanKun has all of these features and a bunch more, although when I say “features” I kind of mean the opposite. I plugged mine in and followed the install instructions. As is pretty typical, this took the form of the plug bringing up its own Wifi access point, the app on the phone connecting to it and sending configuration data, and the plug then using that data to join your network. Except it didn’t work. I connected to the plug’s network, gave it my SSID and password and waited. Nothing happened. No useful diagnostic data. Eventually I plugged my phone into my laptop and ran adb logcat, and the Android debug logs told me that the app was trying to modify a network that it hadn’t created. Apparently this isn’t permitted as of Android 6, but the app was handling this denial by just trying again. I deleted the network from the system settings, restarted the app, and this time the app created the network record and could modify it. It still didn’t work, but that’s because it let me give it a 5GHz network and it only has a 2.4GHz radio, so one reset later and I finally had it online.

The first thing I normally do to one of these things is run nmap with the -O argument, which gives you an indication of what OS it’s running. I didn’t really need to in this case, because if I just telnetted to port 22 I got a dropbear ssh banner. Googling turned up the root password (“p9z34c”) and I was logged into a lightly hacked (and fairly obsolete) OpenWRT environment.

It turns out that here’s a whole community of people playing with these plugs, and it’s common for people to install CGI scripts on them so they can turn them on and off via an API. At first this sounds somewhat confusing, because if the phone app can control the plug then there clearly is some kind of API, right? Well ha yeah ok that’s a great question and oh good lord do things start getting bad quickly at this point.

I’d grabbed the apk for the app and a copy of jadx, an incredibly useful piece of code that’s surprisingly good at turning compiled Android apps into something resembling Java source. I dug through that for a while before figuring out that before packets were being sent, they were being handed off to some sort of encryption code. I couldn’t find that in the app, but there was a native ARM library shipped with it. Running strings on that showed functions with names matching the calls in the Java code, so that made sense. There were also references to AES, which explained why when I ran tcpdump I only saw bizarre garbage packets.

But what was surprising was that most of these packets were substantially similar. There were a load that were identical other than a 16-byte chunk in the middle. That plus the fact that every payload length was a multiple of 16 bytes strongly indicated that AES was being used in ECB mode. In ECB mode each plaintext is split up into 16-byte chunks and encrypted with the same key. The same plaintext will always result in the same encrypted output. This implied that the packets were substantially similar and that the encryption key was static.

Some more digging showed that someone had figured out the encryption key last year, and that someone else had written some tools to control the plug without needing to modify it. The protocol is basically ascii and consists mostly of the MAC address of the target device, a password and a command. This is then encrypted and sent to the device’s IP address. The device then sends a challenge packet containing a random number. The app has to decrypt this, obtain the random number, create a response, encrypt that and send it before the command takes effect. This avoids the most obvious weakness around using ECB – since the same plaintext always encrypts to the same ciphertext, you could just watch encrypted packets go past and replay them to get the same effect, even if you didn’t have the encryption key. Using a random number in a challenge forces you to prove that you actually have the key.

At least, it would do if the numbers were actually random. It turns out that the plug is just calling rand(). Further, it turns out that it never calls srand(). This means that the plug will always generate the same sequence of challenges after a reboot, which means you can still carry out replay attacks if you can reboot the plug. Strong work.

But there was still the question of how the remote control works, since the code on github only worked locally. tcpdumping the traffic from the server and trying to decrypt it in the same way as local packets worked fine, and showed that the only difference was that the packet started “wan” rather than “lan”. The server decrypts the packet, looks at the MAC address, re-encrypts it and sends it over the tunnel to the plug that registered with that address.

That’s not really a great deal of authentication. The protocol permits a password, but the app doesn’t insist on it – some quick playing suggests that about 90% of these devices still use the default password. And the devices are all based on the same wifi module, so the MAC addresses are all in the same range. The process of sending status check packets to the server with every MAC address wouldn’t take that long and would tell you how many of these devices are out there. If they’re using the default password, that’s enough to have full control over them.

There’s some other failings. The github repo mentioned earlier includes a script that allows arbitrary command execution – the wifi configuration information is passed to the system() command, so leaving a semicolon in the middle of it will result in your own commands being executed. Thankfully this doesn’t seem to be true of the daemon that’s listening for the remote control packets, which seems to restrict its use of system() to data entirely under its control. But even if you change the default root password, anyone on your local network can get root on the plug. So that’s a thing. It also downloads firmware updates over http and doesn’t appear to check signatures on them, so there’s the potential for MITM attacks on the plug itself. The remote control server is on AWS unless your timezone is GMT+8, in which case it’s in China. Sorry, Western Australia.

It’s running Linux and includes Busybox and dnsmasq, so plenty of GPLed code. I emailed the manufacturer asking for a copy and got told that they wouldn’t give it to me, which is unsurprising but still disappointing.

The use of AES is still somewhat confusing, given the relatively small amount of security it provides. One thing I’ve wondered is whether it’s not actually intended to provide security at all. The remote servers need to accept connections from anywhere and funnel decent amounts of traffic around from phones to switches. If that weren’t restricted in any way, competitors would be able to use existing servers rather than setting up their own. Using AES at least provides a minor obstacle that might encourage them to set up their own server.

Overall: the hardware seems fine, the software is shoddy and the security is terrible. If you have one of these, set a strong password. There’s no rate-limiting on the server, so a weak password will be broken pretty quickly. It’s also infringing my copyright, so I’d recommend against it on that point alone.

comments