BenchmarkXPRT benchmarks presentation to EBL-WG

2012
HDXPRT Development Community
BenchmarkXPRT benchmarks presentation
to EBL-WG
May 2013
2013

2012
HDXPRT Development CommunityRoadmap for presentation
• Introduction to Principled Technologies, Inc.
• The BenchmarkXPRT Development
Community
• HDXPRT 2012
• TouchXPRT 2013
• WebXPRT 2013
• MobileXPRT (final name under discussion)
• Questions and answers
2BenchmarkXPRT Development Community | www.BenchmarkXPRT.com

HDXPRT Development CommunityAbout Principled Technologies (PT)
• BenchmarkXPRT Community Administrator
• Leading provider of fact-based marketing and
technology assessment services
• Founded by Mark Van Name and Bill Catchings
‒ Over 25 years experience working together on technology
assessment
‒ As journalists, published over a thousand articles on a wide
array of technology subjects
‒ Created the Ziff-Davis Benchmark Operation (ZDBOp), which
developed Winstone, 3D WinBench, WebBench, and many
other benchmarks
‒ Co-founded and/or led several technology assessment firms

HDXPRT Development CommunityWhy more benchmarks?
• The BenchmarkXPRT benchmarks share common
defining characteristics:
‒ Easy-to-understand results
‒ Relatable scenarios
• Community model
‒ Keeps us in touch with what people do in the real world
‒ Why not open source? We need to control derivative
works to avoid benchmark manipulation.

HDXPRT Development CommunityThe BenchmarkXPRT Development Community
• The BenchmarkXPRT Development Community is a
membership-based forum where interested parties can
contribute to the ongoing evolution of the XPRT family of
benchmarks. Members play a key role in ensuring that
these benchmarks incorporate the latest in user-
experience benchmarking and meet the needs and
interests of a range of individuals and organizations.
‒ This is an evolution of the HDXPRT development community,
created in November 2010. All current members are
grandfathered into the new community.
‒ Additional new benchmarks made the old HDXPRT-centric name
obsolete.

HDXPRT Development CommunityBenchmarkXPRT source code
• Our goals for publishing the source code:
‒ Increase transparency in the benchmarking process
‒ Stimulate knowledge transfer to the benchmark community
• Enable innovation and accelerate improvements to benchmarking process
• Provide meaningful results to consumers and OEMs
‒ Opportunity for interested community members to participate in the
development process
• Guide the design specification
• Participate in actual improvement of the code base for all the XPRT
benchmarks
• Give a grounded basis for understanding the actual behavior of the
benchmarks on target systems
• Ground rules:
‒ Source code is available only to BenchmarkXPRT community members.
‒ Members must agree to a license agreement that prevents them from
releasing the code to the public.

HDXPRT Development CommunityBenchmarkXPRT resources
• BenchmarkXPRT Web site
‒ News
‒ White papers
‒ Member forum
‒ Blog
‒ Online database of results
‒ Benchmark-specific Web pages for HDXPRT, TouchXPRT, and
WebXPRT
• Videos
• Twitter
• Facebook
• Presence on SlideShare, YouTube, and LinkedIn

HDXPRT Development CommunityWhat’s HDXPRT?

HDXPRT Development CommunityWhat’s HDXPRT? (cont.)
• HDXPRT, the High Definition Experience & Performance Ratings
Test, evaluates the capabilities of PCs in consumer digital media
usages.
• HDXPRT was the first XPRT benchmark. HDXPRT 2012 is the latest
version.
• HDXPRT uses real, commercially available applications.
• Runs on Windows® 7, 64-bit and Windows 8, 64-bit only.
• Includes tests for popular consumer usage scenarios, providing both
performance scores for each scenario and an overall performance
score.
• Minimum system requirements are an Intel dual-core 2.0GHz
processor or equivalent with 2 GB of RAM, 40 GB of free disk space
and 1,024 x 768, 24-bit color video display settings.
• Minimum supported operating system is Microsoft® Windows 7 64-
bit edition (Language: US English).

HDXPRT Development CommunityWhat’s in HDXPRT 2012?
• HDXPRT 2012 has five use-case categories:
‒ Media Organizer
‒ Media Creator
‒ Photo Blogger
‒ Video Producer
‒ Music Maker

HDXPRT Development CommunityWhat’s in HDXPRT 2012? (cont.)
• HDXPRT 2012 has nine applications:
‒ Adobe® Photoshop® Elements 10
‒ Adobe Premiere® Elements 10
‒ Apple® iTunes® 10.5.2.11
‒ Audacity® 2.0
‒ CyberLink MediaEspresso 6.5
‒ CyberLink PowerDirector 10
‒ GIMP 2.6.11
‒ HandBrake 0.9.5
‒ HDRsoft Photomatix Pro 4.1.4

HDXPRT Development CommunityHDXPRT 2012 scoring
• Scoring includes runtimes for individual usage models, as well
as an overall score.
• The overall score is the geometric mean of the ratios between
test system scores and scores of a calibration system.
• The Calibration system uses an Intel® Pentium® Processor
E6800 processor with 4 GB of 800 MHz DDR2 RAM and a
Seagate® Barracuda 1 TB 7200 RPM SATA drive.
• To see how HDXPRT 2012 compares a sample of Intel i3, i5,
and i7 processors, check out the HDXPRT 2012 scaling paper
at
http://www.principledtechnologies.com/hdxprt/HDXPRT2012
_Scaling_0912.pdf
• Learn more at HDXPRT.com.

HDXPRT Development CommunityHDXPRT 2012 scoring

HDXPRT Development CommunityWhat’s next for HDXPRT?
• We’re working on HDXPRT 2013.
‒ The most significant feedback we’ve received is that HDXPRT
2012 is too big and takes too long to run.
‒ We are working to make HDXPRT 2013 small enough to
download and shorten the running time.
• The scenarios in HDXPRT 2013 are Media Creator, Video Producer,
and Music Maker.
• The 6 applications in HDXPRT 2013 are Apple® iTunes® 11.0.2,
Adobe Photoshop® Elements 11, Audacity® 2.0.3, CyberLink
MediaEspresso 6.5, CyberLink PowerDirector 11, HandBrake/X264
0.9.8.
‒ We believe we can reduce the size and running time without
hurting the value of the benchmark.

HDXPRT Development CommunityWhat’s TouchXPRT?

HDXPRT Development CommunityWhat’s TouchXPRT? (cont.)
• TouchXPRT provides a fair and consistent basis for
evaluating and comparing the capabilities of both
Windows 8 and Windows RT devices.
• TouchXPRT supports both Intel-based and ARM-
based devices.
• Unlike HDXPRT, which uses commercially available
applications, the TouchXPRT scenarios are native
custom code.
• TouchXPRT runs in the Windows 8 Modern UI
(Metro) as a native app.

• TouchXPRT 2013 comprises five scenarios:
‒ Beautify Photo Album
‒ Prepare Photos for Sharing
‒ Convert Videos for Sharing
‒ Export Podcast to MP3
‒ Create Slideshow from Photos
• Since the release of the community preview in October 2012,
TouchXPRT 2013 CP1 And TouchXPRT 2013 have been used to
review and characterize numerous Windows RT and Windows
8 touch-enabled devices.
‒ First appearance was Anandtech’s Surface review, October 23
http://www.anandtech.com/show/6385/microsoft-surface-review/10

• The source code for TouchXPRT 2013 is available to
the community.
‒ We encourage the community to critique the scenarios and
scoring, and to make suggestions for improvement.
‒ In the future, we hope that community members will contribute
scenarios to TouchXPRT.
• Development of future versions of TouchXPRT is ongoing.
If you have feedback on TouchXPRT 2013 and suggestions
for the next version, please let us know.

HDXPRT Development CommunityTouchXPRT Scoring
• For each scenario, TouchXPRT gives two scores: the
elapsed time in seconds and the rate. The rate allows for
“bigger is better” presentations.
‒ For the Beautify Photo Album, Prepare Photos for Sharing, and
Create Slideshow from Photos scenarios, the rate is given as
photos/sec.
‒ For the Convert Videos for Sharing and Export Podcast to MP3
scenarios, the rate is ratio of the length of the clip divided by
the time required for the conversion. For example, an 85.2-
second clip converted in 33.9 seconds is presented as 2.5X.
• Although the community preview did not provide an
overall score, the released version of TouchXPRT 2013
does include an overall score.

HDXPRT Development CommunityTouchXPRT Scoring

HDXPRT Development CommunityWhat’s WebXPRT?

HDXPRT Development CommunityWhat’s WebXPRT? (cont.)
• WebXPRT is a processor-, OS-, and browser-agnostic benchmark that allows you to
compare the performance of almost any Web-enabled device. We have
successfully run this community preview on devices as diverse as the Kindle® Fire,
Android® phones, iPad®, Macs®, and Windows PCs.
• WebXPRT uses standard HTML5 and JavaScript to implement representative Web-
user scenarios. This reliance on standards allows it to run on a wide variety of
devices.
• Unlike some other benchmarks in this class, WebXPRT uses scenarios that real
people can relate to. WebXPRT 2013 comprises four separate workloads:
‒ Photo Effects
‒ Face Detect
‒ Stocks Dashboard
‒ Offline Notes
• Since the release of the community preview in December 2012, WebXPRT 2013
CP1 And WebXPRT 2013 have been used to review and characterize numerous
web-enabled devices.
‒ WebXPRT’s first appearance in a review was in Anandtech’s Acer W510 review
on December 20. http://www.anandtech.com/show/6522/the-clover-trail-
atom-z2760-review-acers-w510-tested

HDXPRT Development CommunityWhat’s WebXPRT? (cont.)
• In addition to the scores, WebXPRT also gives a report on
the HTML 5 capabilities of your device.
• Because it runs in the browser and runs on such a wide
variety of devices, WebXPRT is limited in the amount of
other information it can automatically capture about the
test device.
• Because WebXPRT is based on PT Web servers, some
information about the tests, such as the IP address, will
be exposed to PT servers.
‒ WebXPRT does not keep any personally identifying information.
You can see the list of fields WebXPRT collects at
http://www.principledtechnologies.com/benchmarkxprt/webxp
rt/2013/data-collected

HDXPRT Development CommunityWebXPRT scoring
• Each scenario reports its elapsed time in milliseconds and
a 95 percent confidence interval.
• The rollup score is the geometric mean of the ratios
between test system scores and scores of a calibration
system.
• The WebXPRT 2013 calibration system is an iPad 3
running iOS 6.
• The score calculation excludes outliers. For a full
explanation of how WebXPRT 2013 determines outliers
and calculates its results, see
http://principledtechnologies.com/benchmarkxprt/white
papers/2013/WebXPRT-2013_calculation.pdf.

HDXPRT Development CommunityWebXPRT scoring (cont.)
• Because this is a Web-based test and there may be
some run-to-run variation, WebXPRT 2013 repeats
each scenario seven times.
• To capture any variation, WebXPRT reports the 95
percent confidence interval for the run. Using an iPad
4 scoring 180 +/- 2 as an example, the odds that the
score for any single iteration would be between 178
and 182 are 95 percent.
• We have successfully run WebXPRT 2013 on a wide
variety of devices.

HDXPRT Development CommunityWebXPRT scoring (cont.)

HDXPRT Development CommunityWhat’s MobileXPRT

HDXPRT Development CommunityWhat’s MobileXPRT? (cont.)
• Formerly known as PhoneXPRT
• A new benchmark for evaluating Android-based
devices (Android 4.1 and up)
• Contains a mix of user experience and performance
tests using custom code rather than applications
• Currently testing preliminary versions
• More details over the next few weeks

HDXPRT Development CommunityWhat’s next?
• Continue the development of the existing
benchmarks by adding new tests and features
• Increase community involvement
• Look for other places where there’s need
• Let us know what you’d like to see!

HDXPRT Development CommunityWhat we would like you to do
• Join the community (if not already a member).
• Encourage others to join.
• Take advantage of the design documents, betas, and
Community Previews.
• Send us your suggestions for improvements or new
functionality as soon as possible.
• Collaborate with us as we continue development of
the 2013 and 2014 releases of the benchmarks.
• Participate in the community and spread the word!

Thanks to all Community members for your
participation in the BenchmarkXPRT
development process!
31
We invite all non-members to join the
community to help with the ongoing
evolution of the XPRT benchmarks.
Go to www.benchmarkxprt.com to register.
BenchmarkXPRT Development Community | www.BenchmarkXPRT.com

32
Q & A
BenchmarkXPRT Development Community | www.BenchmarkXPRT.com

BenchmarkXPRT benchmarks presentation to EBL-WG

Recommended

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to BenchmarkXPRT benchmarks presentation to EBL-WG (12)

More from Principled Technologies (20)

Recently uploaded (20)

BenchmarkXPRT benchmarks presentation to EBL-WG