Closed Bug 855554 Opened 11 years ago Closed 11 years ago

mozharness should properly deal with tar wildcards on Ubuntu OR use hg to obtain the talos bits

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

All
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Assigned: jyeo)

References

Details

(Whiteboard: [mozharness][talos])

Attachments

(4 files)

16:51:51     INFO - Running command: ['tar', 'zx', '--strip-components=1', '-f', '/builds/slave/test/build/talos-4a0e7514c66c.tar.gz', '**/talos/'] in /home/cltbld/talos-slave/talos-data
16:51:51     INFO - Copy/paste: tar zx --strip-components=1 -f /builds/slave/test/build/talos-4a0e7514c66c.tar.gz **/talos/
16:51:52     INFO -  tar: Pattern matching characters used in file names
16:51:52     INFO -  tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
16:51:52     INFO -  tar: **/talos: Not found in archive
16:51:52     INFO -  tar: Exiting with failure status due to previous errors
16:51:52    ERROR - Return code: 2
16:51:52    FATAL - Halting on failure while running ['tar', 'zx', '--strip-components=1', '-f', '/builds/slave/test/build/talos-4a0e7514c66c.tar.gz', '**/talos/']


Adding --wildcards fixes the problem, but I'm not sure if this parameter is supported by other platforms (fedora 12 does).
How does this bug block bug 820243? I think this was more for the Ubuntu switch over, no?
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #1)
> How does this bug block bug 820243? I think this was more for the Ubuntu
> switch over, no?

It shouldn't. Blocking bug 713055 is enough.
No longer blocks: 820243
Assignee: nobody → yshun
Attachment #756577 - Flags: feedback?(aki)
Attachment #756578 - Flags: feedback?(aki)
09:30:06     INFO - rmtree: /home/cltbld/talos-slave/talos-data/talos
09:30:06     INFO - retry: Calling <bound method Talos._get_revision of <mozharness.mozilla.testing.talos.Talos object at 0xfe58d0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0xe4af10>, '/builds/slave/talos-slave/test/build/talos_repo'), kwargs: {}, attempt #1
09:30:06     INFO - Running command: ['hgtool.py', '-r', '4a0e7514c66c', 'http://hg.mozilla.org/build/talos', '/builds/slave/talos-slave/test/build/talos_repo']
09:30:06     INFO - Copy/paste: hgtool.py -r 4a0e7514c66c http://hg.mozilla.org/build/talos /builds/slave/talos-slave/test/build/talos_repo
09:30:06    ERROR - caught OS error 2: No such file or directory while running ['hgtool.py', '-r', '4a0e7514c66c', 'http://hg.mozilla.org/build/talos', '/builds/slave/talos-slave/test/build/talos_repo']

The linux machines don't have hgtool? Maybe should use 'vcs': 'hg' instead?
Using hg instead of hgtool is one fix.
Another is to get build/tools onto the box first via hg (or grabbing hgtool via wget), and then using hgtool for talos, but I'm not sure how much we win with that setup.
using hg instead of hgtool
Attachment #756644 - Flags: feedback?(aki)
(In reply to Jason Yeo [:jyeo] from comment #7)
> Created attachment 756644 [details] [diff] [review]
> use hg instead of hgtool
> 
> using hg instead of hgtool

I pushed it to ash-mozharness.
It seems that it is checking out talos but then it has trouble unzipping the dirtyDB.zip (for Mac):
talos-r4-lion-039:talos-slave cltbld$ pwd
/Users/cltbld/talos-slave
talos-r4-lion-039:talos-slave cltbld$ ls -l talos-data/
total 8
-rw-rw-r--  1 cltbld  staff  2368 30 Mar  2011 count_and_reboot.py

https://tbpl.mozilla.org/php/getParsedLog.php?id=23641123&tree=Ash&full=1#error0
11:30:13     INFO - Downloading addons...
11:30:13     INFO - Downloading http://build.mozilla.org/talos/profiles/dirtyDBs.zip to /Users/cltbld/talos-slave/test/build/dirtyDBs.zip
11:30:13     INFO - retry: Calling <bound method Talos._download_file of <mozharness.mozilla.testing.talos.Talos object at 0x10bcd1c10>> with args: (u'http://build.mozilla.org/talos/profiles/dirtyDBs.zip', u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'), kwargs: {}, attempt #1
11:30:13     INFO - Downloaded 7765237 bytes.
11:30:13    FATAL - Can't run command ['unzip', '-q', '-o', u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'] in non-existent directory '/Users/cltbld/talos-slave/test/../talos-data/talos'!
Comment on attachment 756577 [details] [diff] [review]
use hgtool instead of depending on tar

>-class Talos(TestingMixin, BaseScript):
>+class Talos(TestingMixin, BaseScript, VCSMixin):

You can use VCSScript (or MercurialScript) instead of BaseScript + VCSMixin.  It's just shorthand.
Either way works.

>@@ -335,29 +336,32 @@ class Talos(TestingMixin, BaseScript):
>         c = self.config
>         talos_url = self.query_talos_url()
>         if not c.get('webroot') or not talos_url:
>             self.fatal("Both webroot and talos_url need to be set to populate_webroot!")
>         self.info("Populating webroot %s..." % c['webroot'])
>         talos_webdir = os.path.join(c['webroot'], 'talos')
>         self.mkdir_p(c['webroot'], error_level=FATAL)
>         self.rmtree(talos_webdir, error_level=FATAL)
>-        tarball = self.download_file(talos_url, parent_dir=self.workdir,
>-                                     error_level=FATAL)
>-        if self._is_windows():
>-            tarball = self.query_msys_path(tarball)
>-        command = c.get('webroot_extract_cmd')
>-        if command:
>-            command = command % {'tarball': tarball}
>-        else:
>-            tar = self.query_exe('tar', return_type='list')
>-            command = tar + ['zx', '--strip-components=1', '-f', tarball,
>-                             '**/talos/']
>-        self.run_command(command, cwd=c['webroot'],
>-                         error_list=TarErrorList, halt_on_failure=True)
>+
>+        # clone talos' repo
>+        talos_repo_path = os.path.join(self.workdir, 'talos_repo')
>+        repo = {
>+            'repo': 'http://hg.mozilla.org/build/talos',
>+            'vcs': 'hgtool',
>+            'dest': os.path.join(self.workdir, talos_repo_path),
>+            'revision': self.talos_json_config['global']['talos_revision']
>+            }
>+        self.vcs_checkout(**repo)
>+
>+        src_talos_path = os.path.join(talos_repo_path, 'talos')
>+        dest_talos_path = os.path.join(self.workdir, 'talos')
>+        mv_command = ['mv', src_talos_path, dest_talos_path]
>+        self.run_command(mv_command, cwd=self.workdir)

a) maybe use self.move() instead of run_command(['mv' ...])?
b) I don't think you're actually populating c['webroot'], which will be problematic when you actually run the tests.
Attachment #756577 - Flags: feedback?(aki) → feedback+
Comment on attachment 756644 [details] [diff] [review]
use hg instead of hgtool

You'll get this for free if you use MercurialScript.
But this is a valid way of doing it.
Attachment #756644 - Flags: feedback?(aki) → feedback+
Attachment #756578 - Flags: feedback?(aki) → feedback+
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #9)
> It seems that it is checking out talos but then it has trouble unzipping the
> dirtyDB.zip (for Mac):
> talos-r4-lion-039:talos-slave cltbld$ pwd
> /Users/cltbld/talos-slave
> talos-r4-lion-039:talos-slave cltbld$ ls -l talos-data/
> total 8
> -rw-rw-r--  1 cltbld  staff  2368 30 Mar  2011 count_and_reboot.py
> 
> https://tbpl.mozilla.org/php/getParsedLog.
> php?id=23641123&tree=Ash&full=1#error0
> 11:30:13     INFO - Downloading addons...
> 11:30:13     INFO - Downloading
> http://build.mozilla.org/talos/profiles/dirtyDBs.zip to
> /Users/cltbld/talos-slave/test/build/dirtyDBs.zip
> 11:30:13     INFO - retry: Calling <bound method Talos._download_file of
> <mozharness.mozilla.testing.talos.Talos object at 0x10bcd1c10>> with args:
> (u'http://build.mozilla.org/talos/profiles/dirtyDBs.zip',
> u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'), kwargs: {}, attempt #1
> 11:30:13     INFO - Downloaded 7765237 bytes.
> 11:30:13    FATAL - Can't run command ['unzip', '-q', '-o',
> u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'] in non-existent
> directory '/Users/cltbld/talos-slave/test/../talos-data/talos'!

This is probably since we're not populating c['webroot'] anymore (see comment 10).
Attachment #756853 - Flags: feedback?(aki)
Comment on attachment 756853 [details] [diff] [review]
Actually populates webroot

>+        cp_command = ['cp', '-rf', dest_talos_path, talos_webdir]
>+        self.run_command(cp_command, cwd=self.workdir)

I think self.copytree() does what you want.
Hopefully this works for you.
Attachment #756853 - Flags: feedback?(aki) → feedback+
(In reply to Jason Yeo [:jyeo] from comment #13)
> Created attachment 756853 [details] [diff] [review]
> Actually populates webroot

I've pushed it to Ash and re-trigger talos other jobs.
I'm seeing jobs turning green for mac builds. This should be fixed right?

https://tbpl.mozilla.org/?tree=Ash&jobname=talos
If it's green on all platforms, including ubuntu :) , then yes.
We have the crash on Ubuntu and we have hg weirdness on Windows.
Summary: mozharness should properly deal with tar wildcards on Ubuntu → mozharness should properly deal with tar wildcards on Ubuntu OR use hg to obtain the talos bits
Green on all platforms. Fixed.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: Platform Support → Mozharness
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: